| Notes |
The What Works Marketplace
Helping Leaders Use Evidence to Make Smarter Choices
By Alex Neuhoff, Simon Axworthy, Sara Glazer, and Danielle Berfond
Collaborating to accelerate social impact April 2015
Invest in What Works Policy Series
This report is part of Results for America’s “Invest in What Works Policy Series,” which provides ideas and supporting research to policy makers to drive public funds toward evidence-based, results-driven solutions. The series includes policy reports, expert roundtable discussions, and public events. Bloomberg Philanthropies graciously provided support for the launch of the “Invest in What Works Policy Series.”
About Results for America
Results for America (www.results4america.org), launched in April 2012, will improve outcomes for young people, their families, and communities by shifting public resources toward programs and practices that use evidence and data
to improve quality and get better results. Beyond scaling individual programs, Results for America seeks to drive broader systems change so that “investing
in what works” becomes the new norm for allocating public dollars. Results for America has also launched Moneyball for Government, a campaign to encourage governments at all levels to increase their use of evidence and data when investing limited taxpayer dollars.
About The Bridgespan Group
The Bridgespan Group (www.bridgespan.org) is a nonprofit adviser and resource for mission-driven organizations and philanthropists. Bridgespan collaborates with social sector leaders to help scale impact, build leadership, advance philanthropic effectiveness, and accelerate learning. Through its work, Bridgespan focuses on issues related to society’s most important challenges and breaking cycles of intergenerational poverty. Bridgespan’s services include strategy consulting, leadership development, philanthropy advising, and developing and sharing practical insights.
This report has been published with the generous support of Bloomberg Philanthropies
2
Table of Contents
Executive Summary....................................................................................................................................5 Key findings.....................................................................................................................................................................6 Recommendations.................................................................................................................................................. 7 Moving forward...........................................................................................................................................................8
Introduction .............................................................................................................................................................. 9 A marketplace focused on learning and continuous improvement.........9 Methodology ..................................................................................................................................................................11 Defining a common language.................................................................................................................11
Key Finding #1—Supply: A Long Supply Chain..................................... 13 Many steps ...................................................................................................................................................................... 13 Many players................................................................................................................................................................. 13 Many clearinghouses ......................................................................................................................................... 17
Key Finding #2—Demand: A Diverse Group of
Users and Needs .......................................................................................................................................... 20
Several different types of users ........................................................................................................ 20
A variety of needs for different users,
but some key consistencies...................................................................................................................... 21
Key Finding #3—Market Gaps: A Market with
Growing Pains....................................................................................................................................................22
A growing market heading in the right direction....................................................... 22 Six gaps that remain in the market...............................................................................................23
Gap 1: Comprehensiveness...............................................................................................................................23 Gap 2: Implementation........................................................................................................................................25 Gap 3: Guidance.........................................................................................................................................................28 Gap 4: Synthesis ......................................................................................................................................................... 31 Gap 5: Usability .......................................................................................................................................................... 33 Gap 6: Awareness.....................................................................................................................................................35
3
Recommendations: Opportunities to Strengthen
the Market.................................................................................................................................................................37
Strengthen supply................................................................................................................................................ 37
1. Increase the number of studies on interventions available ......................................... 37
2. Increase the amount of information on interventions available..............................39
3. Increase the types of reviews available, not only reviews of
single interventions..........................................................................................................................................39
Build demand............................................................................................................................................................40
1. Increase awareness of sources for evidence on effectiveness,
particularly through existing networks.........................................................................................40
2. Reduce barriers to use of clearinghouses ...................................................................................42
3. Guide decision makers through the selection process....................................................42
Develop infrastructure....................................................................................................................................43
1. Establish common standards ..................................................................................................................43
2. Increase coordination among suppliers ....................................................................................... 44
3. Build a vibrant adviser market .............................................................................................................. 44
Conclusion............................................................................................................................................................... 46 Acknowledgments.....................................................................................................................................48
Appendices.............................................................................................................................................................49 Appendix 1: List of organizations interviewed................................................................49
Appendix 2:
Appendix 3: Appendix 4: Appendix 5: Appendix 6: Appendix 7:
Current efforts to develop and inform
a common evidence framework................................................................. 50
Landscape of clearinghouses.......................................................................... 51 Web analytics for information sources..............................................56 US-focused clearinghouses by domain.............................................59 Comprehensiveness of clearinghouses.............................................62
Intervention-specific information captured systematically on clearinghouses............................................................. 64
4
Executive Summary
As a nation, we owe it to all citizens to invest our resources in the most effective solutions to the problems we face. This is particularly true when it comes to support- ing vulnerable children, families, and communities—an area of both tremendous need and expense that is critical to the health and growth of our nation.
Unfortunately, we cannot say with confidence today that we are making the most of what we spend. Less than 1 percent of federal government spending is backed by even the most basic evidence of impact.1 It may be that many government programs are working. We just don’t know.
The truth is that it is not easy to identify the most effective solutions. Relatively few interventions have been rigorously evaluated; even fewer are proven to have positive results. In fact, most evaluations show mixed results. Therefore, determining whether an intervention works is not black and white and depends on how one plans to use it. Moreover, the needs and populations in our country are constantly changing, so solutions must continually evolve and consider local context.
But we still must embrace an approach based on meaningful data, quality evidence, and rigorous evaluation. A healthy market for evidence on effectiveness would give decision makers—policy makers, funders, and practitioners—the information they need to select the appropriate solution for their circumstances. It also would shed light on areas where there is currently not enough evidence
on effectiveness, and thus more innovation or evaluation is needed.
The purpose of such a market is learning and continuous improvement. It is not the separation of interventions into two neat categories of those that work and those that don’t work. Our research shows that effectiveness is far more nuanced and constantly evolving, and that the required level of effectiveness depends on the application. Judging interventions could discourage innovation because of fear of negative repercussions.
To better understand what it takes for leaders to use evidence in making critical decisions, Results for America, in consultation with The Pew-MacArthur Results First Initiative (Results First)2, commissioned The Bridgespan Group to study this “market” for evidence on effectiveness—specifically, what is the current state of the market? What can be done to strengthen it? And who must lead it?
1 Robert Balfanz, The Power of a Penny: Building Knowledge to Invest in What Works in Education, Invest in What Works Policy Series, Results for America, http://results4america.org/wp-content/ uploads/2014/07/RFAPolicyBrief_k.pdf.
2 The Pew-MacArthur Results First Initiative is a project of The Pew Charitable Trusts and the
John D. and Catherine T. MacArthur Foundation. The opinions expressed in this report are those of the authors and do not necessarily reflect the views of the Pew-MacArthur Results First Initiative, The Pew Charitable Trusts, or the John D. and Catherine T. MacArthur Foundation.
5
Key findings
To better understand how well the market is working today and how it could be strengthened, we completed over 80 interviews with individuals on both
the supply and demand sides.3 On the demand side, we targeted our interviews mostly on two particular domains: child welfare and K–12 education. We then combined the insights from those interviews with secondary research and analysis of supply side information sources to deepen our understanding and test our findings and learnings.
When we began our research, it seemed the exclusive suppliers of information were the online clearinghouses, the demanders of that information were programmatic decision makers, and the commodity they were exchanging was information about interventions that have been proven to be effective— evidence-based practices.
What we found was a market significantly more complex. On the supply side,
we found a long and fragmented supply chain with many more information suppliers than anticipated. On the demand side, we found several types of users with different needs, but among key decision makers demand for evidence is still limited. In fact, we determined that decision makers are only one subset of users for this type of information. Additionally, there are organizations like Pew and Results for America supporting both the supply and demand sides.
The good news is, both supply and demand for evidence on effectiveness appear to be growing. The bad news is, there are growing pains. We identified six gaps in the marketplace that prevent supply from effectively meeting the requirements of demand:
• Gap 1: Comprehensiveness. Decision makers want information on a broader range of interventions with varying levels of effectiveness. They also want to know which interventions have not been reviewed or rated.
• Gap 2: Implementation. Decision makers want information about interventions beyond evidence of impact—including peer experience implementing the intervention—to help them make informed decisions. Few clearinghouses provide this level of information.
• Gap 3: Guidance. Decision makers are looking for guidance and support in selecting and planning to implement the appropriate intervention. Clearing- houses, however, are not set up to provide this, and the intermediaries in this space are still relatively limited.
3 See the Methodology section and Appendix 1 for more detail on interviews and methodology.
6
• Gap 4: Synthesis. Decision makers are looking for more than just interventions. They also are looking for information on policies and management decisions, as well as synthesized findings and best practices. This information is not available systematically and can be difficult to find, even where it does exist.
• Gap 5: Usability. Users do not find clearinghouses easy to use, nor do they understand the differences between them.
• Gap 6: Awareness. Decision makers receive information about interventions from purveyors and peers, but they do not receive information about evidence in a systematic or effective manner.
Recommendations
To address these gaps and strengthen the growing market for evidence on effectiveness, we developed three sets of recommendations. First, we need to strengthen the supply of evidence on effectiveness. Second, we need to build demand for this information. Finally, we need to develop infrastructure that will coordinate and support the interaction between supply and demand.
Within these broad categories, we offer nine specific recommendations:
STRENGTHEN SUPPLY
BUILD DEMAND
DEVELOP INFRASTRUCTURE
1. Increase the number of studies on interventions available
2. Increase the amount of information on interventions available
3. Increase the types of reviews available, not only reviews of single interventions
1. Increase awareness of sources for evidence on effectiveness, particularly through existing networks
2. Reduce barriers to use of clearinghouses
3. Guide decision makers through the selection process, including connecting them
with advisers
1. Establish common
standards
2. Increase coordination
among suppliers
3. Build a vibrant adviser market
These recommendations represent our initial conclusions on how to improve the market. Given that the market is still in an early stage of development, with many small actors and limited resources, it is not always readily apparent who should take on each recommendation. We hope these recommendations provide a starting point for discussion among the many actors who have a stake in making this market function well.
7
Moving forward
Implementing the recommendations in this report will require efforts from all involved in the market. Clearinghouses will need to play a central role as the primary aggregators of evidence on effectiveness. The sponsors and funders of clearinghouses will need to support improvements, such as through allocation of additional resources, revised mandates, or simply advice and encouragement. However, even an ideal set of clearinghouses will not be sufficient to change behavior and ensure the use of evidence on effectiveness. Other players in
the field must support and complement their efforts. In particular, the federal government will need to lead the national conversation about evidence on effectiveness and support the many other actors who play important roles in this market. Researchers and synthesizers will need to work with clearinghouses to make the right information about interventions available. They should also continue to use the available evidence to further improve the pool of effective interventions. Foundations need to direct and support the use of evidence on effectiveness through their grant making and advocacy. Finally, state and local leaders will need to work alongside all of these actors by actively engaging in the market for evidence on effectiveness and using this information to make informed decisions.
Given the large number of actors and relatively limited resources in this market, collaboration and coordination will be essential. It will be important to focus
on continuous improvement, and not on final judgments about what works. Most importantly, it will be critical to stay focused on our end goals: a healthy market for evidence on effectiveness, greater investment in the most effective solutions, and ultimately, better outcomes for vulnerable children, families,
and communities.
8
Introduction
As a nation, we owe it to all citizens to invest our resources in the most effective solutions to the problems we face. This is particularly true when it comes to supporting vulnerable children, families, and communities—an area
of both tremendous need and expense that is critical to the health and growth of our nation.
Unfortunately, we cannot say with confidence today that we are making the most of our spending. Less than 1 percent of federal government spending is backed by even the most basic evidence of impact.4 It may be that many government programs are working. We just don’t know.
Investing in more effective solutions will not be easy. For starters, there is significant inertia in the way that resources are spent today. Making changes
will require political will to shift the funding as well as new skills among those who implement the changes. Additionally, it is not easy to identify what the most effective solutions are. Relatively few interventions have been rigorously evaluated; even fewer are proven to have positive results. In fact, most evaluations show mixed results. Therefore, determining whether something works is not black and white and depends on how one plans to use it. Furthermore, the needs and populations in our country are constantly changing, so our solution set must continually evolve and consider local context.
A marketplace focused on learning and continuous improvement
We must embrace an approach that focuses on learning and continuous improvement and is based on meaningful data, quality evidence, and rigorous evaluation. A healthy market for evidence on effectiveness is critical for two primary reasons. First, it gives decision makers—whom we define as policy makers, funders, and practitioners—the information they need to select the appropriate solution for their circumstances. Second, it sheds light on areas where there is currently not enough evidence on effectiveness and thus more innovation or evaluation is needed.
The purpose of such a market is continuous improvement. It is not the separation of interventions into two neat categories of those that work and those that
4 Robert Balfanz, The Power of a Penny: Building Knowledge to Invest in What Works in Education, Invest in What Works Policy Series, Results for America, http://results4america.org/wp-content/ uploads/2014/07/RFAPolicyBrief_k.pdf.
9
The purpose of a ‘what works’ market is continuous improvement. It is not the separation of interventions into those that work and those that don’t work.
don’t work. Our research shows that effectiveness is far more nuanced (including dimensions such as evaluation rigor, impact, and replicability) and constantly evolving, and that the required
level of effectiveness depends on the application. Placing too much emphasis on judging interventions could discourage innovation, as new interventions by definition do not have evidence. Judging interventions could also discourage the sharing of negative, mixed, or null results for fear of negative repercussions.
Spotlight: Moneyball for Government Part 3—Continuous Learning
Several leading organizations are using data, evidence, and evaluation to learn how to improve outcomes. Watch this video to see more about the work of three such organizations—City Year, Communities in Schools, and LIFT.
For more information: http://moneyballforgov.com/moneyball-for-government- video-series/
An approach that emphasizes learning and continuous improvement depends
on many factors, most notably the ability to learn from past evaluations. But such evidence is not readily accessible today. Moreover, we don’t know enough about who is looking for information or what are they looking for. Are they finding
what they need? From what sources? To begin answering these questions,
and ultimately help increase the flow of information, Results for America, in consultation with The Pew-MacArthur Results First Initiative (Results First)5, commissioned The Bridgespan Group to study this “market” for evidence on effectiveness. We use the term market because the ultimate objective is to connect and align the creators of evidence (or supply—e.g., clearinghouses, evaluators, nonprofits) with the decision makers (or demand—e.g., policy makers, practitioners, private and public funders).
The following sections outline the findings of this study on the market for evidence on effectiveness. We describe the complexity of this market—a long and fragmented supply chain (Key Finding #1) and several types of users with different needs (Key Finding #2). While recognizing that the market is growing and improving, we then identify six gaps in the marketplace that prevent supply from effectively meeting the requirements of demand (Key Finding #3). Finally, we describe nine specific opportunities to address these gaps and strengthen the growing market for evidence on effectiveness (Recommendations).
5 The Pew-MacArthur Results First Initiative is a project of The Pew Charitable Trusts and the
John D. and Catherine T. MacArthur Foundation. The opinions expressed in this report are those of the authors and do not necessarily reflect the views of the Pew-MacArthur Results First Initiative, The Pew Charitable Trusts, or the John D. and Catherine T. MacArthur Foundation.
10
Methodology
To better understand the market for evidence on effectiveness, we gathered input from sources (supply), users (demand), and experts familiar with both.
We conducted in-depth research in two domains—K–12 education and child welfare—to ensure we understood the full complexity of the demand side.
We chose these domains because they are both critical areas for families and communities but also differ significantly with regards to who makes decisions, how funding flows, and the prevalence of effective interventions. We interviewed more than 20 education and child welfare decision makers with various levels of responsibility (state, county, district, and local/school). We also interviewed 10 officials in different government roles, such as the mayor’s office.
When examining the supply side, we broadened our research beyond child welfare and education to include all domains. We focused particularly on clearinghouses, which, for this report, we define as: information sources that aggregate, standardize, review, and rate the evidence base of interventions, acting as repositories that provide input into the decision-making process. However, we did complete a broad scan of the full landscape of information sources in the United States. We also identified a subset of 14 clearinghouses
for more in-depth research and interviews.6 This subset includes those that are larger and more well-known, but also covers a variety of domains and includes both public and private clearinghouses. These 14 will be referenced to a greater extent throughout the report. Finally, we conducted 11 additional interviews with leaders of other information sources, including two international clearinghouses, five purveyors (developers or providers of social programs or interventions), and four advisers and researchers.
While focused on supply and demand, we also interviewed 10 experts with broader perspectives of the market dynamics. In total, we completed more than 80 interviews; the full list of organizations interviewed is available in Appendix 1.
Defining a common language
Our research revealed there is not a common language to discuss the topic of effectiveness. Terms like “evidence-based practices,” “evidence-based programs,” and “what works” are often used but seldom defined or differentiated. One purveyor noted, “If anything can help in this space, it would be adopting a national standard on what the term ‘evidence-based’ means.”
6 Blueprints for Healthy Youth Development; California Evidence-Based Clearinghouse for Child Welfare; Child Trends LINKS; Clearinghouse for Labor Evaluation and Research; Coalition for Evidence-Based Policy; CrimeSolutions.gov; FindYouthInfo.gov; Home Visiting Evidence of Effectiveness; National Registry of Evidence-based Programs and Practices; Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide; Promising Practices Network; Teen Pregnancy Prevention; Washington State Institute for Public Policy; and What Works Clearinghouse
11
There are two particular areas of inconsistency in discussing what works.
This first is the definition of what is being evaluated. People refer to both “evidence-based practices” and “evidence-based programs.” Some use these terms interchangeably. Others give them different meanings, distinguishing practices as components of programs. Still others use the terms to mean other things. Given that both practices and programs can be evaluated, for this report we use the term intervention to refer generally to both practices and programs.
Additionally, there is confusion about the level of evidence required for an intervention to be deemed effective or something that works. First, there is not always a clear distinction between the rigor of the evaluations (i.e., how confident one can be in the results) and the size of the impact attributed to the intervention. We found that information sources had different standards for these dimensions, and some only looked at either rigor or impact.
Further compounding this confusion is the growing (and justified) importance attributed to the replicability of an intervention, or the ability of an intervention to be implemented with fidelity and demonstrate effectiveness more than once. Some are beginning to acknowledge the importance of replicability in rating interventions, but there are not yet consistent standards for integrating it.
We also found that some information sources make an overall judgment as to whether or not an intervention works. This can be confusing, as each source
has its own standards for determining if something works, and thus a single intervention can receive conflicting categorizations. In addition, a clear distinction that something does or does not work can rarely be made. Most evaluations and interventions show mixed results and are highly dependent on the population and outcomes targeted, as well as contextual factors. Moreover, this type of stark distinction can be harmful if it discourages people from sharing negative, mixed, or null results. Similarly, it can discourage innovation, as all new interventions begin with no evidence. The right level and type of evidence needed ultimately depends on the application of the intervention. For example, those focused on continuous improvement must consider all evidence and results upon which to build.
A promising path out of this confusion may lie in the development of a common evidence framework. Such efforts are underway both within and outside of
the government, including the recently revised Education Department General Administrative Regulations (EDGAR); the recently announced Common Guidelines for Education Research and Development; the Cross-Agency Platform for Judging Evidence; a proposed Framework for Continuous Evidence Building; and the Nesta Standards of Evidence in the UK (see details in Appendix 2). In particular, EDGAR creates the opportunity to apply a tiered-evidence framework across all discretionary grant programs to reward evidence-based programs and produce rigorous evaluation of what works in education.
Given that so many efforts are underway, we do not select a specific framework for effectiveness for use in this report. Rather we use a broad definition of evidence on effectiveness when examining the market, which includes information about the rigor of evaluation, impact size, and replicability.
12
Key Finding #1—Supply: A Long Supply Chain
Many steps
Supplying evidence on effectiveness is not a simple matter. To compile
and share evidence, an intervention must first be studied—ideally several times—to determine whether it produces positive outcomes for the target population. Those studies must then be collected, reviewed, and validated by an independent party to ensure the methodology is sound and the outcomes hold across studies. Interventions then need to be rated, or placed along the spectrum of effectiveness, and made available to decision makers and others in a usable format. At this point in the process, it also can be valuable to identify common elements across interventions through synthesis. Decision makers then must be guided on how to select and implement the appropriate interventions based on the available evidence. This process forms a chain for the supply of evidence on effectiveness (see Figure 1).
For this report, we focused on the supply chain steps from the completion of a study through implementation. While developing interventions and studying them are critical steps, we focused primarily on the market for information related to evidence, rather than the creation of interventions and evidence.
Figure 1: Supply Chain for Evidence on Effectiveness
Develop
• Design and • develop intervention with clear intended outcomes •
• Implement intervention in one or more com- munities
Study
Conduct objective study of im- plemented intervention
Evaluate outcomes • and effec- tiveness
of imple- mented intervention
Share
Ensure study
is made available/ accessible to others
Aggregate Standardize
Review Rate
Review and • Rate or rank •
Synthesize
(not neces- • sary for interven-
tion use)
Synthesize findings across inter- ventions
Extract best practices
or themes
Guide selection
Help guide • intervention selection
Guide imple mentation
Help guide or actually implement intervention
Include links to training or imple- mentation resources, etc.
Many players
•
•
•
Aggregate • Standardize
all studies intervention
conducted presentation,
for an inter- including
vention descrip-
Use tion, target
literature population,
searches and study and/or findings
nominations • Categorize Extract and intervention summarize or outcome key findings type
from each • Enable study search and
comparison across inter- ventions
•
• Evaluate intervention outcomes and effec- tiveness
• Provide evidence summary
Submit
study
to peer- reviewed publication • or clearing- house
•
Describe rationale for rating
Given the many steps along the supply chain, it’s no surprise that there are many sources of information (see Figure 2 for definitions and examples). When we
use the terms supply or sources, we mean sources for evidence on effectiveness, which include clearinghouses and others that disseminate and communicate this
validate quality and evidence standard of studies
interventions based on strength of evidence • and effec- tiveness in achieving outcomes •
•
Provide tools or guidance for users to find relevant interven- tions
Include cost estimates, funding strategies, etc.
Focus of report
•
Summarize • effective- ness of practices across inter- ventions
•
13
information to decision makers. While this landscape of information sources may seem crowded, most of these sources play unique and valuable roles.
Starting at the beginning of the supply chain, the purveyors include anyone attempting to expand the use of an intervention. They clearly have an interest, albeit biased, in sharing information about the evidence on their interventions. A purveyor may be the developer of an intervention, such as BELL (Building Educated Leaders for Life), or another party responsible for replication and technical assistance.
The researchers who evaluate interventions—such as Mathematica Policy Research—often make the results of evaluations available to others.
Peers also share interventions with which they have experience. This can be through informal peer-to-peer interactions, as well as more structured forums (e.g., Teachers Pay Teachers in education).
Clearinghouses primarily collect, standardize, and validate evidence on effectiveness. For this report, a clearinghouse is an information source that aggregates, standardizes, reviews, and rates the evidence base of interventions. It acts as a repository that provides input into the decision-making process (e.g., What Works Clearinghouse or Blueprints for Healthy Youth Development).
Clearinghouses make information available to other sources, including the synthesizers who look across multiple interventions and translate the research into language and implications relevant to decision makers. We found that most synthesizers focus on a specific domain (e.g., Casey Family Programs focuses on child welfare).
Advisers help support and guide decision makers through the selection and implementation of effective interventions. We found both public (e.g., Evidence- based Prevention and Intervention Support Center) and private (e.g., Hanover Research) entities playing this role. Some organizations even play multiple roles in providing information, such as Communities in Schools, which acts as both a purveyor of its dropout prevention approach and an adviser to its affiliates.
14
Figure 2: Landscape of information sources—supply
Type of Source
Description
Primary Supply Chain Steps
Examples in Education
Examples in Child Welfare Chain Steps
Purveyors
• Providers or developers who offer products or services to decision makers and often help establish the evidence base
• Develop • Study
• Communities in Schools
• BELL (Building Educated Leaders
for Life)
• Multisystemic Therapy
Services
• Functional Family Therapy LLC
Researchers
• Researchers or evaluators who study the evidence base of an individual program, a particular practice, or a question related to social outcomes
• Study
• Share
• Synthesize
• Mathematica Policy
Research
• Center for Research on Education Outcomes
• Child Trends • Mark Lipsey
Clearinghouses
• Information sources that compile and validate the evidence for interventions
• Aggregate • Standardize • Review
• Rate
• What Works Clearinghouse
• Best Evidence Encyclopedia
• California Evidence-Based Clearinghouse
• Blueprints for Healthy Youth Development
Peers
• Formal or informal networks of peer decision makers who have experience with effective interventions
• Includes professional associations, which provide resources and platforms for sharing experiences and best practices
• Guide selection
• American Association
of School Administrators
• Teachers Pay Teachers
• Child Welfare League of America
• LinkedIn child welfare professional group
Synthesizers
• Organizations, networks, or research centers that synthesize existing research and/ or recommend best practices
• Synthesize
• National Reading
Association
• The New Teacher
Project
• Casey Family Programs
• Chapin Hall
Advisers
• Organizations, individuals, or networks who
assist decision
makers in selecting
or implementing effective interventions
• Includes technical assistance providers and more informal advisers
• Guide selection
• Guide imple-
mentation
• Area Education Agencies
• Hanover Research
• Evidence-based Prevention and Intervention Support Center
• National Resource Centers
15
To help decision makers navigate the large and complicated landscape of information sources, organizations are creating lists or guides of the various websites. For example, the Substance Abuse and Mental Health Services Administration is building A Guide to Evidence-Based Practices that lists available resources by topic area (including clearinghouses and other sources). Several other organizations provide similar lists, including think tanks (e.g., Social Work Policy Institute’s Evidence-Based Practices Registries page) and state agencies (e.g.,
New York State Office of Children and Family Services’ Links to Effective Programs and Practices, and the Ohio Office of Criminal Justice Services’ Evidence-Based Programs and Practices list). While such lists are useful, they are also a sign of how difficult it can be to navigate the existing information on evidence on effectiveness. Additionally, the lists often don’t help to clarify the type or purpose of each source.
Sharing completed impact studies is a critical step in the supply chain, as
it ensures that information compiled about the evidence of interventions is available for all future steps. Currently, there are a large number of both publicly and privately funded impact studies, particularly due to grants that require and provide resources for such evaluations. However, we found that these studies are not always shared publically, which is a breakpoint in the chain. This is often a side effect of the unclear standards for evidence. In particular, the lack of a spectrum or different tiers for interventions can discourage sharing. We heard ambivalence about sharing evidence that’s negative or ambiguous when it was unclear how an intervention would be portrayed. One purveyor admitted it is not on a clearinghouse yet because “you can choose when to be rated, so we chose not to until we can get an A rating.” Uncertainty about negative portrayal can have this type of chilling effect on those who might otherwise be willing to participate in and share evaluations.
Kathy Stack, the advisor for Evidence-Based Innovation at the US Office of Management and Budget, told us that incentives, particularly from the federal government, could help reduce this chilling effect: “The federal government can be a real leader in providing incentives for people to do research and publish findings even when they are negative. We need to change the value system. It’s
‘‘The federal government can be a
real leader in providing incentives for people to do research and publish findings even when they are negative. We need to change the value system.’’ KATHY STACK, ADVISOR, EVIDENCE-BASED INNOVATION, US OFFICE OF MANAGEMENT
AND BUDGET
useful to know what doesn’t work. We need to push for people to preregister studies. This puts it out in the ether that this work is being done.”
We also heard concerns about
the quality of studies being shared. While there are some efforts to create guidelines for study development,
a few people mentioned the lack of
well-defined or well-known standards for design, completion, and documentation. This is believed to lead to poor quality studies. One interviewee also described how the lack of common standards around evidence can cause researchers to put a positive spin on the way that study results are reported. By not clearly
16
and consistently acknowledging the importance of interventions with mixed or null results, there can be rather significant consequences on the front end of the supply chain.
Many clearinghouses
Given our desire to help more
decision makers use evidence, we
are particularly interested in the
role of clearinghouses, which we
found to be the primary sources
for compiling and disseminating
evidence on effectiveness (see Figure 2). While they represent only a portion of the overall supply chain, clearinghouses are fundamental to the ultimate selection and use of interventions.
Clearinghouses essentially assess the rigor of evidence for and impact of an intervention—a task that few decision makers have the capacity or capabilities to do. Such an assessment requires identifying and sifting through all research studies for an individual intervention, evaluating whether this research is valid, and then determining whether the research proves the intervention is effective in achieving certain outcomes.
Using our definition of a clearinghouse, we identified a large and crowded landscape. We reviewed 36 US-focused websites as part of this work (see Appendix 3), including:
• 15 clearinghouses primarily run and funded by federal, state, or local governments;
• 9 clearinghouses primarily funded by governments but run independently; and • 11 nongovernment clearinghouses.
We also identified a sample of 15 international clearinghouses, primarily UK-based, which can serve as reference points for their US counterparts (see Spotlight: The United Kingdom’s What Works Network and Appendix 3).
Within the US landscape, there are some well-known and relatively high-trafficked sites, such as the Department of Education’s What Works Clearinghouse, with approximately 8,000 daily visitors. There are also many less-visited clearinghouses. In fact, even a few of the more well-known clearinghouses cited only between 200 and 700 total visitors per day (see Appendix 4 for web analytic information).
Some US clearinghouses target one or only a few social policy domains, while others cover a broad range of policy domains. For example, the Campbell Collaboration and the Coalition for Evidence-Based Policy look at interventions across several different areas of social programs (see Appendix 5 for comparison of clearinghouses by policy domains).
While they represent only a portion of the overall supply chain, clearinghouses are fundamental to the ultimate selection and use of interventions.
17
Spotlight: The United Kingdom’s What Works Network
The What Works Network is an initiative launched in 2013 by the Government of the United Kingdom. The initiative is based on the principle that good decision making should be informed by robust evidence on what has been shown to work, or not work, in the past.
The Network builds on the successful model of the National Institute for Health
and Care Excellence (NICE), which was set up in the 1990s to inform health spending and clinical decisions by carrying out robust assessments on the impact and cost effectiveness of medical interventions. NICE is now one of the seven independent What Works Centres that make up the Network. The others are: Education Endowment Foundation, What Works Centre for Crime Reduction,
Early Intervention Foundation, What Works Centre for Local Economic Growth,
and two that are in progress—Centre for Ageing Better and What Works Centre
for Wellbeing. Each Centre is funded by a combination of government and nongovernment sources, and the Network is supported by a team in the UK Cabinet Office; however, all of the centres are operationally independent of government.
The ultimate objective is for policy makers, practitioners, and commissioners in these policy areas to make decisions informed by evidence on impact and cost effectiveness, alongside other considerations. This will allow them to use resources as efficiently as possible. As such, all of the centres are working to balance the provision of evidence online with direct practitioner/commissioner engagement
to ensure that this information is actually used on the ground. It is challenging but crucial to reach decision makers directly. While the initiative is still new, and each centre is at a different stage of development, this unique approach should be watched closely and potentially leveraged in other countries.
For more information: https://www.gov.uk/what-works-network
Even within a given domain, there can be multiple relevant clearinghouses.
For example, the California Evidence-Based Clearinghouse for Child Welfare directly targets decision makers in child welfare, but other clearinghouses also cover parts of this domain, including Blueprints for Healthy Youth Development (Blueprints), Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-based Programs and Practices, and others. In education, the What Works Clearinghouse is the largest clearinghouse, but the Best Evidence Encyclopedia and National Dropout Prevention Center/Network are also important sources.
About the proliferation of resources, one clearinghouse interviewee said, “They are all different. They have different goals, policies, procedures, and criteria. These differences aren’t necessarily better or worse. [They] just [have] different purposes.” The breadth or focus of a given clearinghouse is driven by the mission of the organization in which they are situated. For example, FindYouthInfo.gov was developed by the federal Interagency Working Group on Youth Programs
to provide tools and resources for a range of youth-oriented programs.
18
Each clearinghouse has criteria for its own review and rating process. However, these criteria are not consistent across clearinghouses due to the current lack of industry-wide standards.
Each clearinghouse has criteria for its own review and rating process. Most clearinghouses document their criteria on their websites. However, the criteria are not consistent across clearinghouses due to the current lack of industry-wide standards. Some, such as the What Works Clearinghouse, provide ratings on
individual dimensions of evidence (e.g., one rating on the extent of evidence and one rating on effectiveness). Others, like Blueprints, provide a single rating across all dimensions (e.g., Promising Practices is a single rating based on dimensions of intervention specificity, evaluation quality, intervention impact, and dissemination readiness). There is also variation within a given dimension. For example, quality or rigor of evaluation can be based on the type of study (e.g., randomized control trial versus quasi-experimental design) or the number of sites studied. Several additional types of inconsistencies exist, making it difficult to compare across websites.
19
Key Finding #2—Demand: A Diverse Group of Users and Needs
Several different types of users
The end users of evidence on effectiveness are ultimately decision makers. We define decision makers as the policy makers, funders, and practitioners who decide which interventions to fund and implement.
However, we were surprised to find several other types of users—namely, the intermediaries who are also providers of some evidence on effectiveness. They include:
• Advisers who use information from clearinghouses and other sources to guide decision makers in the appropriate selection and use of interventions.
• Researchers and synthesizers who contribute to clearinghouses through
their evaluations. Researchers also use evidence on effectiveness in their
work to expand the pool of effective interventions. They reference these sources as they determine where new interventions are required to fill gaps, and where additional evidence is required to demonstrate effectiveness for a given intervention. Synthesizers sometimes use clearinghouses as sources for interventions and studies that they use in their own syntheses, which are often aimed at enhancing the effectiveness of interventions.
• Purveyors who are primarily interested in seeing whether their interventions are listed on clearinghouses. Purveyors also sometimes use these repositories as learning platforms, seeing what other information on evidence on effectiveness is available in their domains.
Occasionally, even the general public uses clearinghouses to access existing research and best practices for issues of particular concern (e.g., parents of children struggling in school).
In interviews, the clearinghouses acknowledged that they do not target a single audience, although a few do have a more narrow focus. One clearinghouse described its audience as, “Anyone who is in a position to influence how social spending is allocated—anyone in a position to decide how evaluation resources are used. Could be federal, state, local level; could be a philanthropic foundation.”
Even among decision makers, the users of evidence on effectiveness vary, as the level and type of decisions vary significantly by domain and location. For example, in child welfare key decision makers are generally state or county administrators. On the other hand, decision makers in education are often district or school administrators.
20
A variety of needs for different users, but some key consistencies
Across all user types and domains, our interviews identified that certain evidence is widely sought. Everyone is interested in understanding whether certain interventions are effective. In addition, several users across domains and levels expressed their need for synthesized findings and best practices, as well as information and support that could help guide selection of interventions and next steps.
Yet, decision makers in different domains also engage with evidence on effectiveness in different ways and value different information and supports. Market dynamics and characteristics of decision makers shape how certain domains search for and utilize evidence on effectiveness.
For example, in child welfare, decisions can be bucketed into two types:
1) policies or principles around practices in the interactions with children and families (e.g., decisions about removal from the home, caseload); and 2) contracts with third-party providers for services (e.g., case management, foster care). Generally, decisions to changes policies or providers are infrequent due to long bid cycles and are constrained by existing programs and services. As a result, implementation of new interventions and search for evidence on effectiveness are rare. We also found that many of the decision makers have research backgrounds and are looking for detailed scientific information, including raw data and underlying studies.
In education, on the other hand, decisions are made frequently around school and district management (e.g., hiring, schedules, accommodating special populations), and programs and services (e.g., curriculum, teacher professional development, student support services). Hence, the likelihood of searching
for evidence on effectiveness and implementing new interventions is high. Additionally, teachers and administrators have little time to spend reviewing in-depth research on interventions, so they highly value synthesis reports and extractions of best practices over detailed studies. They also have strong peer networks that share their experiences with specific interventions and offer more general counsel.
21
Key Finding #3—Market Gaps: A Market with Growing Pains
A growing market heading in the right direction
While the increasingly diverse landscape of information sources is complex,
we see evidence of a positive trend: both supply and demand for evidence on effectiveness are growing. Decision makers across domains voiced their interest in understanding such evidence and described their searches for this information. Indeed, they are often attempting to integrate this information into funding and intervention selection decisions.
We also see many organizations responding at an incredible pace to decision makers’ need for information. Over the past decade, government at all levels
and other funders have sponsored the creation of new clearinghouses or further development of existing ones in order to validate and make available the existing evidence on effectiveness.
Even within their resource-constrained environments, several clearinghouses described their efforts to continually improve their website functionality and content. By soliciting user feedback, they are starting to identify the gaps in meeting users’ needs and plan their responses. About this improvement process, one clearinghouse interviewee said, “A clearinghouse is a long-term process...
It’s a big cultural change; it’s accelerating now, but it’s not going to change overnight. It’s going to require repetition and getting the incentives set up right.”
Researchers and synthesizers described how they also are responding to demand by further developing and sharing evidence in various forms. And peer networks and advisers are increasingly taking on this topic, disseminating evidence on effectiveness, and guiding decision makers in its use.
Interestingly, we heard that clearinghouses and synthesizers are even influencing their domains to improve evidence on effectiveness through their roles in defining rigorous criteria. “One priority for us is driving funding to fill gaps,
such as getting developers to better codify models and getting actual empirical articles to be more rigorous,” said a representative at one clearinghouse. “For example, the What Works Clearinghouse has an author guideline template, which is getting everyone to follow guidelines and include what needs to be included.” Such guidelines—whether explicit like those from the What Works Clearinghouse7 or implicit—can help increase quality evaluation studies, ultimately expanding and improving the universe of information that can be made available to decision makers.
Budget cuts in recent years have driven decision makers at all levels to pay more attention to evidence as a way to use limited resources more effectively.
7 What Works ClearinghouseTM Reporting Guide for Study Authors, http://ies.ed.gov/ncee/wwc/pdf/ reference_resources/wwc_gsa_v1.pdf.
22
In addition, decision makers are starting to recognize the value of results achieved by effective interventions. This recognition is largely being pushed by the government—at the federal, state, and local levels—encouraging the funding, development, and use of evidence on effectiveness.
Six gaps that remain in the market
As is often the case, with growth comes growing pains. Gaps have emerged in the market between what users want and what suppliers offer. While there is a strong foundation on which to build, these gaps must be addressed for the market to maintain its momentum.
Gap 1: Comprehensiveness. Decision makers want information on a broader range of interventions with varying levels of effectiveness. They also want to know which interventions have not been reviewed or rated.
Those who decide which interventions to pick face a dizzying array of choices. Imagine a county child welfare director who must select an intervention that will provide in-home supports to families whose children have just been reunified after spending time in foster care. The director can select from a handful of well-known national interventions supported by purveyors, another set of codified “do-it-yourself” interventions, and a large number of homegrown interventions. In many instances, there is also the option of continuing with the intervention already being delivered, which could fall into any of the above categories. Similar spectrums of intervention options are available in most domains.
To support informed decisions, comprehensiveness of information sources— particularly clearinghouses—is critical. Being aware of what has limited or mixed evidence is as important as knowing what has strong evidence. For example, a decision maker might assume an intervention not listed on a clearinghouse simply has not been reviewed, when in fact, it may have been found to be ineffective.
Unfortunately, such comprehensive information is not currently available from most clearinghouses (see Appendix 6 for clearinghouse coverage). Clearinghouses play a critical role in identifying effective interventions, using literature reviews to systematically identify studies in their domains. However, this information alone is not enough. Clearinghouses do not typically list interventions that:
• have not yet been considered in their specific domain(s),
• are in the queue for review,
• did not meet minimum standards/eligibility for review,
• were reviewed but not rated (e.g., due to insufficient number or type of studies), • lacked strong research evidence, or
• lacked evidence of intervention effectiveness.
One of the reasons for this lack of comprehensiveness is that clearinghouses can only review and display the studies available to them. Our research revealed
23
a bias in the research community to publish only studies that show positive outcomes. Therefore, many studies on interventions that are currently at other points along the spectrum do not show up in clearinghouses’ literature reviews and will not make it onto the sites.
One government interviewee looked to policy as a potential solution: “Once a state implements a new policy, whatever it is, it should make sure there is an evaluation process attached and publish what it finds—and ensure that findings are widely disseminated. Evaluation and publishing might be done to some extent, but it’s hard to find.” Programs like the US Department of Education’s Innovation (i3) Fund are starting to play this policy role. Such programs help increase the number of studies available by requiring rigorous evaluations and submission of evaluations to clearinghouses. In fact, the Department of Education has funded 117 unique i3 projects to date, all of which are being evaluated and submitted.
A few clearinghouses are much further along in terms of comprehensiveness.
For example, the What Works Clearinghouse is very transparent in listing many studies—including those that are under review and those that did not meet minimum standards (i.e., ‘ineligible for review’)—which are not commonly included on other clearinghouses. CrimeSolutions.gov has also made progress lately, posting an “insufficient evidence” list for interventions that lack strong research evidence. A few other clearinghouses are beginning to follow suit. One clearinghouse interviewee noted, “The next step for us is to figure out a way to show the results of the thousands of studies we review [which are not deemed effective] and to put them out in a way that is balanced and clear. It’s important to have that kind of counterweight, some sort of neutral party reporting on these things.”
However, these are the exceptions. Our interviews revealed the following reasons why most clearinghouses omit studies or interventions that are ineffective or simply inconclusive:
1. Ideology: They believe their core role is to show only those interventions that work, often according to high standards of rigor.
2. Fear of negative consequences: Sites need purveyors to cooperate and so do not want to discourage them from submitting studies for review.
3. Selection risk: They believe decision makers may incorrectly assume that all interventions listed on the site work and thus inadvertently select an ineffective or inconclusive intervention.
4. Lack of resources: To conserve their limited resources, clearinghouses selectively choose to examine interventions that have a strong likelihood of passing criteria.
In our interviews, lack of resources was the most cited constraint. For many clearinghouses, particularly those of the federal government, there are very few dedicated full-time employees and work is primarily conducted by contractors. Limited resources create challenges for clearinghouses, preventing them from holistically fulfilling their core role of building out comprehensive databases of
24
interventions and studies. “We have reviewed 10,000 studies, but there are many more studies out there...we only have so much bandwidth,” one clearinghouse interviewee said.
In fact, several clearinghouses do recognize the importance of more comprehensive databases and would like to address the gaps when they have sufficient resources. However, research and approaches are rapidly changing in many of these domains, and clearinghouses sometimes struggle to keep up- to-date. It may be necessary to frequently re-review particular interventions or continually review newer models in order to remain relevant for decision makers.
Gap 2: Implementation. Decision makers want information about interventions beyond evidence of impact—including peer experience implementing the intervention—to help them make informed decisions. Few clearinghouses provide this level of information.
Evidence on a specific intervention—which clearinghouses provide in the form of underlying studies and validation of research—is important but not sufficient to make informed decisions about adopting an intervention. Like any other consumer, decision makers are making a purchase decision. In addition to evidence on effectiveness, they need to weigh the costs and benefits of the product and the likelihood the product will actually work for them. Therefore, they need more intervention-specific information, as well as examples of other communities to help conceptualize implementation.
Specifically, to help them make purchase decisions, we heard that decision makers need the following information:
• Up-front and ongoing costs of implementation for comparison across interventions.
• A comparison of costs to the likely benefits of intervention success (e.g., positive social outcomes).
• Detailed characteristics of the population addressed by a given study or intervention (e.g., age, gender, ethnicity, special needs).
• The level of intervention modification possible without impacting the evidence base.
• Readiness for dissemination and availability of implementation planning details—such as manuals available, required resources, training available/ required, timelines, and links to the underlying resources.
• Contact information for the purveyor for follow-up questions or purchasing.
We heard about the need for such information throughout our interviews. One principal explained, “We do research outside of our school. We want to know what other high-performing districts or schools of our size are using. For us, it’s important to know size and budget: can I implement with my staff and budget?” In addition, one child welfare administrator noted, “We don’t use [clearinghouses] to make decisions. We believe we need to read the papers themselves; we need
25
to assure ourselves that those studies have been completed on populations that are very similar to ours.”
Yet this information is often difficult to find on the clearinghouses. A primary factor is that evaluators do not capture such data systematically within the intervention studies themselves. A few clearinghouses try to contact the evaluators or purveyors to request this additional detail. However, even when this information is included in the studies, some clearinghouses do not systematically extract and present it in a way that is readily accessible to decision makers (see Appendix 7 for availability by clearinghouse).
Of the different types of data, cost of
the intervention is the most frequently
requested but the most difficult to find,
primarily due to its lack of inclusion in
studies. “The number one thing people
asked about, but could not get, was
information about cost,” said Rebecca
Kilburn, a senior economist at the RAND
Corporation and former director of the
Promising Practices Network (which has
since closed). “For example, someone
might be considering a few programs and might want to go with the one that has lower outcomes if it has even lower relative cost. There are those types of trade-offs they are making [between costs and benefits].”
A few clearinghouses are working to make additional intervention-specific information accessible. The Washington State Institute for Public Policy (WSIPP)
is a well-regarded resource for cost-benefit analysis on interventions for the
state of Washington, and similar work is being initiated by Results First (see Spotlight: Washington State Institute for Public Policy and Results First). WSIPP
is not a standard clearinghouse. Rather, it conducts nonpartisan research at
the direction of the Washington State Legislature. However, it does act in the standard clearinghouse role of aggregating, standardizing, reviewing, and rating interventions, and then goes a step further to determine whether the intervention is a good investment. Other clearinghouses can use WSIPP’s information (as the UK’s Investing in Children clearinghouse already does) or conduct similar analyses. Home Visiting Evidence of Effectiveness (HomVEE) has a user-centered feedback loop to understand what people are looking for on the site. In response to feedback, HomVEE has started to provide more implementation planning support.
‘‘The number one thing people asked about, but could not get, was information about cost. For example, someone might be considering a few programs and might want to go with the one that has lower outcomes if it has even lower relative cost.’’ REBECCA KILBURN, SENIOR ECONOMIST,
RAND CORPORATION AND FORMER DIRECTOR, PROMISING PRACTICES NETWORK
26
Spotlight: Washington State Institute for Public Policy and Results First
Each year, states face tough budget choices, and policy makers need to focus taxpayer dollars on the programs and services that yield the greatest benefits in the most cost-effective ways. Washington State has implemented a unique approach to meeting this challenge. In the mid-1990s, the state legislature began to direct the Washington State Institute for Public Policy (WSIPP)—a nonpartisan research institute—to identify evidence-based policies that have been shown to improve particular outcomes.
Most notably, WSIPP uses a cutting-edge research model to produce independent assessments of the benefits and costs of a wide range of program options from
the perspective of Washington citizens and taxpayers. The results of this approach enable policy makers to compare and rank programs. Such information has been well-received both within and outside of Washington State. While originally focused on criminal justice, WSIPP has applied the same evidence-based and benefit-cost approach to other public policy areas, including K–12 education, early childhood education, child welfare, and mental health.
Based on WSIPP’s model, The Pew-MacArthur Results First Initiative (Results First) is working with a growing number of jurisdictions to help them implement a customized cost-benefit approach to policy making. Results First provides hands-on technical assistance to help decision makers compile and analyze the program, population,
and cost data needed to operate the model. It then helps these jurisdictions interpret the results and make evidence-based budget and policy decisions that provide the strongest return on public investments.
Since 2011, 16 states and four California counties have partnered with Results First to apply this customized, innovative cost-benefit approach to policy and budget decision making. Over the past two fiscal years, five states—Iowa, Massachusetts, New Mexico, New York, and Vermont—have used the Results First model to target $81 million in funding toward programs that the model shows will achieve higher returns for taxpayers. A number of states also have passed statutes embedding Results First cost-benefit analysis into their budget processes.
For more information: http://www.wsipp.wa.gov/BenefitCost and http://www. pewstates.org/projects/pew-macarthur-results-first-initiative-328069
In addition to cost, decision makers highlighted peer experience with a given intervention as another key area of interest. Peer perspectives have been and will continue to be a very important factor across domains. Demonstrating the strength of peer input, the website Teachers Pay Teachers—a marketplace for education resources created by teachers—receives 50–100 times the number of unique monthly visitors as even the most visited clearinghouses (see Appendix 4 for relative estimates of different information sources).
Decision makers want to understand and connect with peers who have implemented the specific interventions they are considering. They generally want to understand the successes and challenges—lessons learned—from peer experiences, and heed their advice. Many decision makers believe that information from peers with
27
similar population characteristics is the best proxy they can get to determine the likelihood of success of a certain intervention with their populations. Not surprisingly, most expressed the desire to know where interventions had been successfully implemented, what lessons have been learned, and how to contact these communities.
However, information on peer experiences is not readily available through clearinghouses or other formal evidence on effectiveness providers. In broad terms, clearinghouses are an information repository; they do not have a sales, support, or tracking function. They do not know who has selected an intervention and whether it was ultimately successful. Moreover, they serve as objective reviewers of evidence to maintain their credibility. They do not offer opinions
on specific interventions. Therefore, while peer implementation may be of high value to decision makers, clearinghouses are not the right vehicle to supply it. Yet, there are no other information sources taking on this role today.
Gap 3: Guidance. Decision makers are looking for guidance and support in selecting and planning to implement the appropriate intervention. Clearing houses, however, are not set up to provide this, and the intermediaries in this space are still relatively limited.
In addition to information, our interviews revealed that many decision makers need more support to help them make informed decisions about which interventions to select. It requires a lot of time and expertise to gather the necessary information for each option and weigh decision factors such as costs, size of outcomes, and likelihood of outcomes. One child welfare administrator pointed out, “I am not certain that the information is not out there; I just think it’s not out there in a format that is easily digestible, easily understandable. In all of my policy staff, I do not have one highly skilled research-type mind.” Under a lot of pressure from stakeholders to make informed, well-researched decisions, decision makers need to be able to substantiate their final selections.
Through our interviews, we heard from many decision makers who want tools to help guide them through the selection process. These include:
• Tools (e.g., surveys) to assess community needs, such as risk factors and required outcomes;
• Criteria or steps (e.g., guides, webinars) to use for selection among potential interventions;
• Ability to sort or filter interventions by multiple dimensions of the target population—assuming this information is captured systematically to begin with; and
• Next steps (e.g., guides, links) to take after intervention selection.
These types of support could be self-administered or involve decision makers relying on advisory services to guide them through the selection process. The level of support required depends on the decision maker, the complexity of
28
selection options, and the type of decision. One educator noted the importance of accessing multiple types of support: “Having a wide web presence is a start, but also having a person who you can easily reach out to in order to get more help would be great.”
Most clearinghouses do not provide sufficient selection guidance. For starters, they are constrained by the information included in the underlying studies. When information on a target population is available, clearinghouses do not always extract this information to make it sortable, filterable, and searchable. Few clearinghouses provide or link decision makers to assessment tools or step- by-step guidance; where such tools do exist, they can be difficult to find on the websites. And almost none of the clearinghouses have the resources to provide hands-on advisory services through the selection process. “My impression is that you do have to go to that individual level,” said Danielle Mason, who heads the What Works Team within the UK Cabinet Office. “[Helping individuals] requires going beyond saying ‘here’s what works’ and saying ‘here’s how it’s applicable
‘‘[Helping individuals] requires going beyond saying ‘here’s what works’ and saying ‘here’s how it’s applicable to you’—but this is a challenge since it
is much more resource-intensive than just providing information.’’
DANIELLE MASON, HEAD, WHAT WORKS TEAM, UK CABINET OFFICE
to you’—but this is a challenge since it is much more resource-intensive than just providing information.”
However, several clearinghouses are recognizing the need to provide greater guidance, and a few are beginning to build out self-administered tools and capabilities. For example, the California Evidence-Based Clearinghouse for
Child Welfare is currently revising its existing section on Screening and Assessment Tools to better meet the needs of its audiences. The National Registry of Evidence- based Programs and Practices allows decision makers to search for applicable interventions by gender, geographic location, age, and race/ethnicity of the intervention target population. The Office of Adolescent Health’s Teen Pregnancy Prevention resource center includes a full page of resources for selecting an evidence-based program.
Advisers are beginning to play a role in guiding decision makers to choose interventions. This may start with needs assessments for their populations, evolve into research and effective intervention selection guidance, and flow all the way through to technical advice in implementation. Advisers are aware of multiple clearinghouses and use them regularly. They leverage evidence in the research they do to support their partner organizations or communities. One adviser explained, “The reason we are tasked with questions, even though the information is available on clearinghouses, is that while the information up there is useful, it tends to be very broad and doesn’t provide information relevant to implementation issues and considerations.”
A number of advisers already operate in this space. For instance, a few education interviewees mentioned Hanover Research, an information services firm that uses a fixed-fee partnership model. Hanover Research conducts custom research
29
projects for its K–12 partner organizations—schools, school districts, and regional education agencies—that involve the review of research and best practices to address a specific question. We also heard about the Evidence-based Prevention and Intervention Support Center (EPISCenter), sponsored by the government
of Pennsylvania, which works closely with communities in the state to select and implement youth-focused interventions from a menu (see Spotlight: EPISCenter, a Center of Excellence). Several universities, such as Case Western Reserve University, are developing similar centers of excellence that serve as hubs for local advisory services. One provider said, “The network of centers of excellence, like the EPISCenter and others, are great examples of intermediaries...they know who to call and their calls will be answered. Not many groups can span these boundaries.”
Spotlight: EPISCenter, a Center of Excellence
The Evidence-based Prevention and Intervention Support Center, or EPISCenter,
a project of the Prevention Research Center at Penn State University, is a unique and successful adviser model. EPISCenter is aimed at providing technical assistance to communities and service providers in Pennsylvania to support the implementation of a menu of evidence-based prevention and intervention programs. The center is
a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency and Penn State University. It receives funding and support from the commission and from the Pennsylvania Department of Public Welfare.
Where communities have received evidence-based program implementation grants from Pennsylvania, experts from the EPISCenter provide technical assistance to local staff on implementation, evaluation, and sustainability, and help develop the infrastructure to monitor the program. Over time, providers build internal capacity for these operations and many continue to report data to the EPISCenter even
after their initial funding has ended. A lighter model for support and resources is available for non-grantee communities. As a center developed by the state and supporting recipients of grants, EPISCenter also can communicate and connect with a variety of stakeholders. Since 2008, the center has assisted in establishing nearly 300 evidence-based programs in more than 120 communities throughout the state.
For more information: http://www.episcenter.psu.edu/
With a more limited set of consumers and more resources than clearinghouses (such as through revenue models), advisers can provide one-on-one interaction and support for decision makers. They have expertise in research, as well as practice translating studies into application. They also tend to be more accessible to decision makers than clearinghouses.
Unfortunately, the adviser market is sparse and underrepresented, and awareness is low. The importance of local context makes most advisers regional in nature, driving fragmentation. In areas without advisers, it is unclear who exactly could or should play this more hands-on role. Due to the high demand for one-on-one support services, and the limited capacity for any given organization to do this in a high-touch way, most regions lack a sufficient number of advisers to meet the needs of decision makers.
30
There are a variety of perspectives on who should fill this gap. Some feel that more university research centers could be hubs for a local advisory function, whereas others believe that state governments should develop centers which provide this support, perhaps associated with grant programs. However, there is no obvious or one-size-fits-all solution. Each state—along with relevant funders— will likely need to develop a plan to build out these capabilities.
Gap 4: Synthesis. Decision makers are looking for more than just interventions. They also are looking for information on policies and management decisions, as well as synthesized findings and best practices. This information is not available systematically and can be difficult to find, even where it does exist.
Through our interviews, it became evident that decision makers face other important decisions in their roles that do not involve selecting a single intervention. They must make important decisions about policies and management—and would like to base these decisions on evidence, as well. For example:
• In education, leaders need to make decisions on school/district management issues (e.g., hiring and firing, budget management, schedules and calendars, accommodating special populations).
• In child welfare, county administrators need to make decisions on policies or principles for management of work (e.g., case load, removal from the home).
Unfortunately, decision makers have few sources to turn to for this information today. Clearinghouses often lack evidence on effectiveness related to policies and management decisions. This most likely is due to a lack of studies or evaluations on these types of practices, as it is harder to establish evidence and objectivity in research. However, it could also partially result from clearinghouses having limited resources, less expertise in these areas, or a lack of clarity on this additional set of decisions that decision makers need to make.
Decision makers also are looking for summarized information about effective interventions, primarily best practices and components of effective interventions. One school district administrator explained, “If we are looking at specific programs or materials, then What Works Clearinghouse is a good place; if we are looking for best practices in certain areas, we will go to universities or other organizations where it is their area of expertise.”
Summarized reports can help translate detailed scientific research into practical guidance and “how tos.” This is particularly helpful for practitioners—audiences such as principals and teachers—who might not have backgrounds in technical research and evidence but are still interested in improving outcomes. This also can be relevant for decision makers who have limited time and bandwidth for detailed research, but who are interested in summaries of key findings or implications.
Due to constraints within their existing infrastructure, decision makers are often looking for summaries of successful model components across interventions. This occurs frequently in child welfare where implementing a totally new model
31
is uncommon due to entrenched services ‘‘We try to avoid only pointing
and provider relationships. Decision makers are looking for incremental or continuous improvement, rather than a complete overhaul to a new packaged intervention. Synthesis reports and meta-analyses
affiliates to programs that have been demonstrated to be effective... We strive to look across the programs, figure out the most effective strategies, and help them incorporate these strategies into what they are already doing.’’ DAN CARDINALI, PRESIDENT,
COMMUNITIES IN SCHOOLS
that identify successful components or
practices across multiple interventions are
useful, as decision makers can implement
these to improve existing work. Dan
Cardinali, president of Communities in
Schools, explained, “We try to avoid only pointing affiliates to programs that have been demonstrated to be effective for lots of reasons: they might be too costly, require intensive training, or can’t be sustained after initial investment. We strive to look across the programs, figure out the most effective strategies, and help them incorporate these strategies into what they are already doing.”
In 2013, there were over 370,000 downloads of What Works Clearinghouse’s practice guides—more than twice the number of downloads of its intervention reports.
Clearly, decision makers value
summarized information and synthesis.
Yet it is unclear who is responsible for providing these things. No one currently does it systematically, although some clearinghouses have started to take on more of a synthesizer role. The What Works Clearinghouse has launched practice guides,
which have been well received. In 2013, there were over 370,000 downloads of the practice guides—more than twice the number of downloads of intervention reports. The What Works Clearinghouse interviewee told us, “For practitioners, our most useful product is the practice guides.” However, most clearinghouses do not play this synthesizer role, nor do they believe it is their role to play.
A number of intermediaries—in particular synthesizers and researchers—provide this support, either by conducting meta-analyses or synthesizing existing research. For instance, Chapin Hall provides additional research and synthesis in the child welfare space. In some cases, intermediaries leverage the information from clearinghouses, and then provide additional value through aggregation and analysis. As a clearinghouse that also acts as a researcher, Child Trends is able
to leverage its own underlying database of over 650 programs to synthesize learnings. The resulting fact sheets, called LINKS Syntheses, are organized by program population, outcome, and approach.
More often, there is not a direct link or relationship between clearinghouses and synthesizers/researchers. This limits the benefits to decision makers. Complicating matters, even when synthesizers directly leverage underlying research that is evidence based, the findings or recommendations of the synthesized research are not necessarily evidence based and statistically significant (outside of meta-analysis). Decision makers therefore need to
32
keep in mind this distinction between effective interventions and synthesis as they use this information.
There are many synthesizers and researchers in each domain. However, it is
not always apparent where decision makers should go for certain information. There are no clear winners or market leaders. Although there are multiple sources for synthesized information, the information is not always sufficient, nor does it necessarily reach decision makers. Clearinghouses are partly playing the role of synthesizers themselves but only on an ad-hoc basis. Synthesizers and researchers are important players, but they are still sub-scale and may not be known to decision makers. A smoother supply chain from interventions to synthesis is required in order to engage decision makers.
Gap 5: Usability. Users do not find clearinghouses easy to use, nor do they understand the differences between them.
We interviewed several types of users with a range of clearinghouse experience. While they acknowledged the critical role of clearinghouses, many were confused or dissatisfied with their experiences—which discourages them from using the sites on a more regular basis.
Part of their confusion is due to a lack of understanding of the differences between clearinghouses. This is particularly vexing when there are several clearinghouses reviewing the same studies for the same domain outcomes (e.g., in child welfare), or even using the same underlying databases. While each of the clearinghouses may add unique value, it is not always apparent what that unique value is. Clearinghouses often operate in isolation and do not clearly articulate their points of differentiation relative to other sites in the space. In interviews, decision makers were unable to identify the differences among clearinghouses.
Different rating scales and criteria are another area of confusion for users. There are often advantages to the differences in validation processes, due to different audiences or outcomes of interest. Yet since the differences are not clearly defined, ratings can appear inconsistent to users. Consider Blueprints, which rates interventions as “model” or “promising” based on intervention specificity, evaluation quality, intervention impact, and dissemination readiness. Meanwhile, the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) Model Program Guide (MPG) rates the same interventions as “effective,” “promising,” or “no effects” based on a program’s conceptual framework, design quality,
‘‘Users have told us it’s confusing. They go to one clearinghouse and there is this rating. They go to another, and it’s a different rating. What does that mean?’’
CAMBRIA ROSE WALSH, PROJECT MANAGER,
THE CALIFORNIA EVIDENCE-BASED CLEARINGHOUSE FOR CHILD WELFARE
outcome evidence, and program fidelity. Therefore, users can find conflicting information about a given intervention across clearinghouses. For example, Big Brothers Big Sisters of America received the highest classification of “effective” from OJJDP’s MPG, but it was only rated “promising” by Blueprints. “Users
33
have told us it’s confusing,” admitted Cambria Rose Walsh, project manager of
the California Evidence-Based Clearinghouse for Child Welfare. “They go to one clearinghouse and there is this rating. They go to another, and it’s a different rating. What does that mean?”
To start addressing this challenge, Results First has developed a Clearinghouse Database aimed at compiling and comparing the ratings of interventions across clearinghouses (see Spotlight: Results First Clearinghouse Database).
Spotlight: Results First Clearinghouse Database
The Pew-MacArthur Results First Initiative (Results First) created the Clearinghouse Database to assist policy makers at all levels of government in making data-driven budget decisions. This one-stop online resource gives users an easy way to find information on the effectiveness of more than 900 interventions as rated by
eight national research clearinghouses. The database uses a simple color-coding system to reconcile the different ratings terminology used by clearinghouses and provides hyperlinks to their program pages so users can easily access the valuable information that has been compiled. Results First plans to enhance this resource in the near future by including additional search options and information.
For more information: http://www.pewtrusts.org/~/media/Assets/2014/09/Results_ First_Clearinghouse_Database.xlsx?la=en
Furthermore, it can actually be difficult for users to determine which clearing- houses relate to the domains in which they are interested. Clearinghouses use
a range of descriptors to indicate the topics they cover (see Appendix 5 for US- focused clearinghouses by domain). For example, many clearinghouses describe their content as covering broad topics such as youth, children and families, or community health. It is unclear how their intervention scope might overlap with those clearinghouses that have more narrowly defined themes (e.g., juvenile justice, teen pregnancy).
Poor and confusing website navigation also causes problems for clearinghouse users. Interviewees did not find the sites to be very intuitive in design, which sometimes prevented them from using the full functionality of the site. Even frequent users often were unaware of certain site content. Others lamented that the sites were overwhelming, and it was unclear how to effectively use them. One child welfare administrator explained, “If I go to a federal website, it might take a half hour to find one piece of data because I have to maneuver through so many different sites [and possibly] be directed somewhere else.”
Our high-level evaluation of the major clearinghouses confirmed many of the concerns of end users. The majority of clearinghouses do not provide adequate information on what to do upon site entry or how to best use the site. Navigation can be overly complex, often causing users to click numerous times to reach desired information. Searchability and sortability on key dimensions such as
34
name, intervention type, outcome, and setting are also lacking, making it hard for users to find the most appropriate intervention.
There are best practices that clearinghouses can learn from. For example, Blueprints provides clear navigation guidance upon site entry as well as video tutorials for users. Meanwhile, the UK’s Early Intervention Foundation Guidebook allows users to search or select along multiple dimensions and defines ratings within the results to ease comprehension. Some larger sites have developed feedback loops with target audiences to improve delivery. For example, OJJDP’s MPG is conducting focus groups with its users to improve site usability.
Part of the difficulty in navigating these sites can be attributed to the broad range of users that clearinghouses are targeting. In trying to be everything to all audiences, clearinghouses can become a bit unwieldy or overwhelming. It is difficult to guide users effectively through the site when each user might be looking for different information in different ways.
Many clearinghouses, therefore, remain difficult to differentiate and difficult to use. This will be important to address moving forward, as audiences need to understand how to use the sites in order to obtain their full benefits.
Gap 6: Awareness. Decision makers receive information about interventions from purveyors and peers, but they do not receive information about evidence in a systematic or effective manner.
Almost all the decision makers we interviewed were aware of the concept of evidence on effectiveness. Most have heard about clearinghouses or other sources in their respective domains, and this awareness appears to be growing. However, many admitted they do not use these resources often and do not appear to be making decisions with evidence on effectiveness top of mind. The most prevalent reason given was that the strong presence of purveyor and word- of-mouth information crowds out evidence on effectiveness, which they would often have to seek out.
Purveyors have strong marketing efforts and relationships, and their presence in many domains competes with evidence for attention. For example, decision makers in education are inundated with vendor pitches that tend to crowd out detailed research. The pitches are voluminous, accessible, and provided directly to decision makers in clear terms.
‘‘I get inundated with products and salespeople constantly. If I would be inundated on the other end—‘here are the evidence-based practices...’—then I would not have to use the other resources.’’
DR. LAURENE LANICH, ASSISTANT SUPERINTENDENT OF TEACHING AND LEARNING, WEST DES MOINES COMMUNITY SCHOOLS
It is difficult for clearinghouses or other evidence on effectiveness information sources to directly compete with the sales pitches of vendors, developers, and providers. Further complicating this imbalance
is that purveyors often tout their products as evidence-based practices, whether or not they have been
35
officially validated. These messages not only guide decision makers toward sub-optimal interventions, but they can also desensitize them to the important concept of evidence. Dr. Laurene Lanich, assistant superintendent of teaching and learning at the West Des Moines Community Schools in Iowa noted, “I get inundated with products and salespeople constantly. If I would be inundated on the other end—‘here are the evidence-based practices, here is how to use this information based on your needs’—then I would not have to use the other resources and that would make my job a lot easier.”
Additionally, decision makers have relied heavily on formal and informal peer networks and word-of-mouth for information regarding effective interventions. Some networks include: professional networks and associations (e.g., American
Decision makers have relied heavily on formal and informal peer networks and word-of- mouth for information regarding effective interventions.
Association of School Administrators, Child Welfare League of America), key conferences, online forums and blogs, and contacts and personal networks.
Clearinghouses and other sources of evidence on effectiveness do currently try to leverage the more formal peer
networks. Most clearinghouse interviewees mentioned having a presence at conferences—as exhibitors or presenters—as their primary method for promoting their websites. Some clearinghouses also provide training for certain professional groups. For example, the What Works Clearinghouse conducted a webinar for faculty of pre-service teacher and principal training programs to showcase their resources for teachers and administrators. Additionally, the College of Policing in the UK, which leads the What Works Centre for Crime Reduction, has established related training programs for practitioners.
However, outside of conferences and occasional trainings, information from peer networks is not always explicitly tied to evidence on effectiveness. It is unclear how strongly or frequently professional networks and associations themselves promote the use of evidence or refer decision makers to clearinghouses or other resources. We have heard that while some networks and groups are well-versed in the topic, and potentially advocate for the use of evidence, others are less friendly to or knowledgeable about the subject matter. Even the sharing of peer experience with effective interventions is done on an ad-hoc basis across these various mechanisms.
If these trends continue, where decision makers are aware of but do not access or act on evidence on effectiveness, it will not matter if the other gaps are fixed and the supply of information is improved. Formal sources for effective interventions will be able to compete against informal structures only if they are more compre- hensive and readily accessible, and if they actively reach out to decision makers.
36
Recommendations: Opportunities to
Strengthen the Market
The market for evidence on effectiveness is complex and changing, and
there are no straightforward solutions for the identified gaps. Furthermore,
it is not only a matter of improving the clearinghouses. Even an ideal set of clearinghouses would not be sufficient to change behavior and ensure the use of evidence, given that this information is only one input to the decision-making process. Strengthening the market for evidence on effectiveness will require efforts by a variety of actors over many years.
The recommendations put forth in this report represent our perspectives on what needs to be done to help address the six gaps identified in the market for evidence on effectiveness. Given that this is still an evolving market, in many cases it is not clear that a single entity is responsible for acting on the recommendations. We have provided our perspective on who should lead for each recommendation, whether clearinghouses, evaluators and other researchers, or federal agencies and other clearinghouse sponsors. We recognize additional discussion and experimentation may be necessary to determine the appropriate actor. With these recommendations, we hope to stimulate further discussion and action among the critical stakeholders.
We also have noted an approximate time frame for each recommendation, to indicate whether we believe the designated actors can take action in the short- or long-term (although it might take longer to see the intended results or impact):
• Shortterm: relatively straightforward recommendation, which can be undertaken in the next one to three years
• Longterm: recommendation that might be challenging to implement and/or have a lot prerequisite activities, and that will require three or more years
Here we put forward three sets of recommended actions: those to strengthen the supply of evidence on effectiveness, those to build demand for this information, and those to develop infrastructure for the market on evidence.
Strengthen supply
1. Increase the number of studies on interventions available
Create a registry of impact studies: Evaluators should be encouraged to register planned impact studies of interventions in a central, shared, public registry, along with key identifying characteristics such as study size, type, population, and timeline. Such a registry would increase the likelihood that study results, including those with ineffective or mixed findings, are shared and make it easier for suppliers and practitioners to find them. The field of medicine provides a strong example of this approach, as all clinical trials are required by law to be registered in order for results to be published.
37
• Main actors: Third-party organizations or federal agencies (create/host registry); evaluators (register studies)
• Supporting actors: Federal/state government grant programs and other funders change grant requirements
• Time frame: Short-term
Make all studies public: Evaluators should make all impact studies, especially ones funded by public dollars, available to clearinghouses and others for review. This would allow clearinghouses to be more comprehensive in what they display. To achieve this, funders—in particular, the government—should require that grantees submit studies to the relevant government or private clearinghouse for review.
• Main actors: Evaluators submit studies for review
• Supporting actors: Federal/state government grant programs and other
funders change grant requirements to require submission
• Time frame: Short-term
Display all studies and interventions: Clearinghouses should include all available studies and known interventions. In addition to interventions
that they judge to be effective, clearinghouses should display reviewed interventions with negative, insufficient, or inconclusive evidence, as well as interventions with unknown evidence (i.e., interventions without any studies or with studies that have not yet been reviewed). Including such information would help to engage users who want to see a comprehensive comparison of interventions. Clearinghouses must be thoughtful in how they post and articulate these additional studies and interventions so that users easily understand the distinction.
• Main actors: Clearinghouses display all studies and known interventions • Time frame: Short-term
Review prioritized studies and interventions: Clearinghouses should try
to review more studies and interventions, prioritizing them based on user demand in order to best use limited resources. Ideally, this method would include directly asking users (current and potential) what topics and outcomes most interest them. This would help increase the relevance of clearinghouses to users. Clearinghouses should also articulate and share this prioritization method on their sites, so the ordering is clear to users.
• Main actors: Clearinghouses review more relevant studies/interventions; prioritize topic areas by user interest
• Time frame: Short-term
38
2. Increase the amount of information on interventions available
Include more intervention detail on clearinghouses: Evaluators should include additional information—about the purveyors, costs, timelines, implementation support, and target population—in intervention studies so that it is readily available to clearinghouses. For this to happen, government grant programs and other funders should require grantees to include this information in their studies. Developers, providers, and communities also need to help ensure this information is collected and made available to the evaluators. Clearinghouses should then systematically capture and display this additional information for the interventions listed on their sites. They should also articulate where data
is not available, as the information (or lack thereof) will help decision makers understand the implications of selecting a particular intervention.
• Main actors: Evaluators include detailed intervention information in studies; clearinghouses display additional information
• Supporting actors: Federal/state government grant programs and other funders change grant requirements; developers/providers collect and share information
• Time frame: Short-term
Develop a system to connect peers: To help decision makers learn from their peers, it would be ideal to develop a way for decision makers to connect with peers who have relevant experiences with a particular intervention. This could happen in a variety of ways, from a publically available database of which interventions communities are using, to an online platform or discussion forum. Whichever form this system takes, it would require some sort of verification
or registration process to ensure it is used appropriately. Additionally, this system should be linked to clearinghouses to allow decision makers to easily navigate between data and peer information. A pilot would likely be necessary in order to determine the best way to connect decision makers to peers without overwhelming already busy practitioners. Given the strong foundation and widespread knowledge of the What Works Clearinghouse, the education field might be a good choice for a pilot. If the pilot is successful, this approach could be expanded so that each domain has a peer connection system.
• Main actors: Third-party organizations host and monitor peer connection systems • Time frame: Long-term
3. Increase the types of reviews available, not only reviews of single interventions
Conduct more metaanalysis: Researchers should evaluate more practices (e.g., types of interventions, model components/characteristics) through systematic reviews or meta-analysis, and make this information available to decision makers both directly and through clearinghouses. Clearinghouses should include or direct decision makers to existing meta-analyses but
should also consider evaluating more practices themselves to appeal more to audiences less interested in specific program models. To do so, clearinghouses
39
could leverage their existing repositories of study reviews. Without additional resources, this might require a trade-off of time for clearinghouse researchers. Also, this shift might not be relevant for all domains, as meta-analysis requires a substantial number of controlled studies from which to draw conclusions.
• Main actors: Researchers conduct more meta-analysis/systematic reviews; clearinghouses conduct and display more meta-analysis/systematic reviews
• Supporting actors: Federal agencies and other clearinghouse funders encourage clearinghouses to reallocate resources to include more meta- analysis and systematic reviews
• Time frame: Long-term
Create more synthesis reports: Synthesizers should continue to develop summary reports or best practice guides about interventions using information from clearinghouses. These reports should be made available to decision makers both directly from synthesizers, and through clearinghouses and other information sources, where appropriate. For example, the Substance Abuse and Mental Health Services Administration (which sponsors the National Registry of Evidence-based Programs and Practices), provides some synthesized information through A Guide to Evidence-Based Practices, which lists available resources by topic area. When resources permit, clearinghouses also should start playing a larger role in creating synthesis reports or guides themselves, leveraging their underlying databases of interventions. This may require a trade-off of time and resources from continuing to review studies and interventions, which each clearinghouse will need to evaluate for itself.
• Main actors: Synthesizers continue to provide synthesis; share with clearing- houses and decision makers; clearinghouses conduct more synthesis; connect decision makers to additional resources, where possible
• Supporting actors: Federal agencies and other clearinghouse funders encourage clearinghouses to reallocate some resources to include or direct decision makers to synthesis
• Time frame: Long-term Build demand
1. Increase awareness of sources for evidence on effectiveness, particularly through existing networks
Educate practitioners about evidence: Education and training programs for practitioners (e.g., pre-service teacher programs) should include guidance on evidence. Practitioners should learn about the importance of using evidence on effectiveness to make decisions. They should also be introduced to resources for information on evidence and trained on how to understand and interpret this evidence. Introducing these concepts and skills to the next generation
of social sector leaders has the potential to create a significant culture shift toward evidence.
40
• Main actors: Practitioner training programs and in-service professional development/executive education opportunities adjust curriculum
• Supporting actors: Clearinghouses, researchers, and synthesizers provide information for training programs
• Time frame: Short-term
Harness the power of peer networks: Informal and formal peer networks, including professional associations and learning communities, should leverage their position as connectors and gatherers to encourage use of evidence and effective interventions. These networks should raise the
topic of evidence more often with their constituents, directing them to the relevant resources, fostering discussions, and encouraging the sharing of experiences. Intermediaries—in particular, advisers—also should increase broad marketing to and education of decision makers. While continuing to target decision makers directly, they should also tap into peer networks to reach decision makers where they are—with information that is presented
in an accessible way. Intermediaries should focus these marketing efforts on increasing awareness about their own role. However, they also should try to increase awareness about the importance of evidence more generally and
the availability of clearinghouses as a resource for evidence on effectiveness. While there is a wide range of existing peer networks, further research might identify a lack of such associations for key decision makers in certain domains and therefore a need to either develop new ones or expand existing ones.
• Main actors: Peer networks encourage discussion and use of evidence on effectiveness; advisers, synthesizers, and researchers create marketing plan and focus on raising awareness through peer networks
• Time frame: Short-term
Engage and target intermediaries: Advisers, synthesizers, and researchers play an important role in reaching the ultimate decision makers and
helping with selection of effective interventions. To the extent possible, clearinghouses should ensure such intermediaries are equipped with the appropriate information to play this role. In order to do so, clearinghouses should identify and market specifically to the intermediaries, engaging them to better understand and integrate their needs and preferences into the websites themselves. Clearinghouse outreach should encourage intermediaries to use clearinghouses as a resource for their own work and guide decision makers to clearinghouses to help in their selection processes. As the intermediary market is still growing, in the near term, clearinghouses should continue their existing efforts to market and reach out directly to decision makers.
• Main actors: Clearinghouses market to and engage key intermediaries
• Supporting actors: Advisers, synthesizers, and researchers provide candid
input to clearinghouses to facilitate improvement
• Time frame: Short-term
41
2. Reduce barriers to use of clearinghouses
Clearly explain role of clearinghouse: Clearinghouses should articulate
their purpose and points of differentiation from other information sources. They should define a specific objective and identify a small number of target audiences, with designations of how each audience should use the website. Clearinghouses should also describe how their websites are different from other information sources (e.g., focused only on interventions with the strongest evidence, looking at different intervention outcomes or target populations). This information should be readily accessible to all users on the website.
• Main actors: Clearinghouses adjust websites to incorporate clear articulation of differentiation
• Time frame: Short-term
Enhance clearinghouse usability: Clearinghouses should update their websites to enhance web design, functionality, and search optimization, incorporating best practices and user feedback. There is also an opportunity for clearinghouses to share with each other their techniques and user feedback— particularly among the federal clearinghouses where a cross-agency mechanism for discussion already exists. In general, clearinghouses should be looking to improve overall usability, such as by: leveraging search engine optimization
to ensure easy location of their websites; ensuring that primary information
is displayed in a general ‘F-shaped pattern’ (which is known to be used when reading web content); providing a search bar in the top right of their websites; and limiting the number of clicks to intervention summary information.
• Main actors: Clearinghouses update websites to incorporate best practices; share best practices with each other
• Supporting actors: Federal agencies and other clearinghouse funders encourage clearinghouses to update web design and functionality; nonprofits provide expertise and technical assistance to clearinghouses for website updates
• Time frame: Short-term
3. Guide decision makers through the selection process
Provide selfguided selection tools: Clearinghouses should include self-guided tools and supports on their websites to help decision makers with selection. In particular, all clearinghouses should provide functionality to sort and filter interventions by multiple dimensions of the interventions’ target population (e.g., age, gender, ethnicity), if they do not have this already. This functionality helps decision makers select interventions relevant for their specific context and needs. Clearinghouses should describe how online tools should be used and what additional research might be necessary to make a decision. Clearinghouses should acknowledge that for many decision makers, such self-guided tools and supports will only be a first step, and it will be necessary to consult advisers for hands-on guidance.
42
• Main actors: Clearinghouses develop additional self-guided tools and supports; in particular, allow users to search by target population
• Supporting actors: Federal agencies and other clearinghouse funders encourage some reallocation of resources for clearinghouses to include more self-guided tools
• Time frame: Short-term
Connect decision makers with adviser market: Clearinghouses should try
to educate decision makers about how advisers can be helpful in providing hands-on selection and implementation supports. They should guide decision makers on how to find advisers in their domains and direct them to any known existing advisers. We recognize government-run clearinghouses
may be constrained in providing direction to external resources. However, clearinghouses should not feel the need to be comprehensive or consider these resources to be referrals.
• Main actors: Clearinghouses provide guidance on adviser market • Time frame: Short-term
Develop infrastructure
1. Establish common standards
Create common evidence standards: Common evidence standards for reviewing and validating studies should be developed in order to create consistent definitions and guidelines across clearinghouses and help alleviate some of the confusion for decision makers. These evidence standards should include a common spectrum of evidence and a common language to articulate different tiers around level of impact, rigor, and replicability of an intervention. Efforts currently underway to define standards for categorizing evidence
and applying it to decision making include the recently announced Common Guidelines for Education Research and Development and the Education Department General Administrative Regulations (EDGAR) (see Appendix 2). Any new effort should work with and build off of these current efforts, likely requiring a public-private partnership.
• Main actors: Public-private partnership facilitates creation process
• Supporting actors: Clearinghouses and other sources of information adhere
to standards
• Time frame: Long-term
Create common standards across the supply chain: Common standards also should be developed across the supply chain, including those for designing and conducting studies, implementing interventions, and providing technical assistance. Common standards at these steps in the supply chain would help increase the quality of studies submitted, make reviewing and comparing studies easier for clearinghouses, and help practitioners more consistently
43
and successfully implement interventions. Additional research is needed to understand what standards are in use today, if any, and to identify the right organization (likely a public-private partnership) to facilitate the development of common standards.
• Main actors: Public-private partnership facilitates standards creation process
• Supporting actors: Evaluators conduct studies according to standards; practitioners implement according to standards; intermediaries provide support according to standards
• Time frame: Long-term
2. Increase coordination among suppliers
Create a coordinating body for clearinghouses: Public and private
entities should partner to create a group that will coordinate activities for clearinghouses and other information sources for evidence on effectiveness. This group should include representatives from each clearinghouse, key decision makers, and intermediaries, as well as a third-party organization
to facilitate. The group should focus on activities such as creating and implementing standards and educating stakeholders about evidence on effectiveness and available resources.
• Main actors: Third-party organization create/facilitate coordinating body
• Supporting actors: Clearinghouses, intermediaries, decision makers
participate in the coordinating body
• Time frame: Long-term
3. Build a vibrant adviser market
Build and expand adviser market: Effective, affordable advisory organizations should be created and expanded to help decision makers select effective interventions. These advisers would ideally work with decision makers to understand their needs, identify potential interventions or solutions, and select the option that best fits their communities. Through our research, we identified several types of actors that currently fill this role, including: public and private universities, which often have departments and/or graduate students with expertise in a specific domain and local context; state or local government- funded centers (e.g., Evidence-based Prevention and Intervention Support Center in Pennsylvania); existing resource centers, which can provide more hands-on support (e.g., Regional Educational Laboratories, National Resource Centers); and for-profit firms for communities that are able to allocate resources for support. Other examples of effective models of advisers likely exist. Given the growing interest in evidence on effectiveness and the clear need for additional supports, this market is ripe for innovation and growth.
44
• Main actors: Universities, state and local governments, resource centers, and for-profit firms create or further develop advisory function
• Supporting actors: Funders support development of advisers to spur use of evidence
• Time frame: Long-term
Include funds for selection support: In recent years, several funders have started providing grants that require grantees to select from a list of effective interventions. In such instances, funders should ensure grantees have the support they need to select the intervention that is the best fit for them. This may include funding for the grantee’s time spent systematically evaluating options or to pay for an adviser.
• Main actors: Federal/state government grant programs and other funders include funds in grants to support selection
• Time frame: Long-term
45
Conclusion
The market for evidence on effectiveness will never
be straightforward due to the diverse needs of users. However, in an ideal world, we can envision a robust supply chain that both helps to ensure decision makers get the right amount and type of evidence to inform their intervention decisions, and helps to improve and expand the pool of effective interventions. The recommendations in this report can move us in this direction by substantially improving the supply chain and better connecting it to decision makers’ needs.
We believe it is important to start by implementing fixes toward the beginning
of the supply chain. Ideally, evaluators and purveyors would share all completed studies with the relevant clearinghouses. Clearinghouses consolidate and validate studies for all interventions in their domains and make information about interventions at all parts of the evidence spectrum available to users in a clear and usable format. Clearinghouses then provide or direct users to intermediary resources, where information is synthesized into best practices. Researchers and others use the intervention evidence and synthesis to develop new interventions and improve the effectiveness and evidence-base of others.
For decision makers, clearinghouses would provide user-friendly websites with information and tools to guide selection of interventions. An integrated third- party site then connects decision makers to peers to learn about implementation of interventions. More importantly, clearinghouses and other information sources direct decision makers to a vibrant adviser market where they can receive hands- on support. Advisers also proactively reach out to decision makers to encourage them to think about evidence on effectiveness. They also leverage clearinghouses and other information sources to guide decision makers in selecting interventions that are appropriate for their contexts and can be implemented with fidelity. Researchers then evaluate these intervention implementations, which leads to
a virtuous cycle of evidence refinement and development, further strengthening the supply chain.
The market for evidence on effectiveness has progressed impressively in recent years and is moving in the direction of the ideal world described above. In particular, the clearinghouses are doing great work at making evidence on effectiveness available to a variety of users and filling what was previously a significant disconnect in the supply chain. We identified the six supply chain
46
gaps to help clearinghouses and other information sources continue to develop and improve.
Implementing the recommendations in this report will require efforts from all involved in the market. Clearinghouses will need to play a central role as the primary aggregators of evidence on effectiveness. The sponsors and funders
of clearinghouses will need to support improvements, such as through allocation of additional resources, revised mandates, or simply advice and encouragement. However, even an ideal set of clearinghouses will not be sufficient to change behavior and ensure the use of evidence on effectiveness. Other players in
the field must support and complement their efforts. In particular, the federal government will need to lead the national conversation about evidence
and support the many other actors who play important roles in this market. Researchers and synthesizers will need to work with clearinghouses to make
the right information about interventions available. They also should continue to use the available evidence to further improve the pool of effective interventions. Foundations need to direct and support the use of evidence on effectiveness through their grantmaking and advocacy. Finally, state and local leaders will need to work alongside all of these actors by engaging in the market for evidence on effectiveness and using evidence to make informed decisions.
The recommendations from this study could help make strong progress towards the ideal market for evidence on effectiveness by filling the existing gaps in the supply chain. However, our recommendations focus mainly on the market for information and by themselves will not lead to increased use of evidence. The realignment of major funding sources is necessary to support interventions that are achieving results and building their evidence. And the market will not grow unless interventions continue to be created, evaluated, and refined.
Given the large number of actors and relatively limited resources in this market, collaboration and coordination will be essential. It also will be important to focus on continuous improvement and not on final judgments about what works.
Most importantly, it will be critical to stay focused on our end goals: a healthy market for evidence on effectiveness, greater investment in the most effective solutions, and ultimately, better outcomes for vulnerable children, families, and communities.
Alex Neuhoff is a partner in The Bridgespan Group’s New York office. Simon Axworthy is a manager, Sara Glazer is a case team leader, and Danielle Berfond is a consultant, all working out of the New York office.
47
Acknowledgments
Michele Jolin and Shivam Mallick Shah at Results for America and The Bridgespan Group authors would very much like to thank the many leaders and organizations that agreed to participate in interviews, which were the primary inputs to this report. We appreciate them sharing their experiences and insights.
We also appreciate the thoughtful guidance for this project provided by the Pew-MacArthur Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation, particularly from Karen Lyons and Gary VanLandingham. Finally, as an initiative of America Achieves, Results for America also wants to thank Bloomberg Philanthropies for its generous support for the research, production, and release of this paper.
We also wish to acknowledge others who provided additional time and input, including Jon Baron, Kelly Fitzsimmons, Rob Ivry, Jim Shelton, and Kathy Stack.
48
Appendices
Appendix 1: List of organizations interviewed
DEMAND
SUPPLY
EXPERTS
Education
Child Welfare
Crosscutting Government
• Boston Preparatory Charter Public Schools
• Charlotte- Mecklenburg Schools (NC)
• Duval County
Public
Schools (FL) • Fayette
County Public Schools (KY)
• Jefferson County
Public
Schools (KY)
• Kansas
Board of
Regents
• NewYork
State Education Department
• Officeofthe State Super- intendent (NC)
• Palisades Park
Elementary
School (NC)
• Urbandale
Community
Schools (IA)
• Wellesley
Middle
School (MA)
• WestDes
Moines Community Schools (IA)
• Alabama Department of Human Resources
• Guiford County DSS (NC)
• Iowa Department of Human Services
• Iowa Children’s Justice, State Court Adminis- tration
• LosAngeles County
Department of Mental Health (CA)
• North Carolina DSS
• Placer County
Health and Human Services (CA)
• Wake County DSS (NC)
• ChildWelfare Program,
National Conference of State Legislatures
• Cityof Kansas City, MO
• CityofLos Angeles, California
• Colorado Governor’s Budget Office
• Education Program,
National Conference of State Legislatures
• Florida Senate
Committee on Ways and Means
• FiscalAffairs, National
Conference of State Legislatures
• Social Innovation
& Civic Participation, The White House
• USOfficeof Management and Budget
• BELL
• BlueprintsforHealthyYouth
Development
• CaliforniaEvidence-Based
Clearinghouse for Child
Welfare
• ChildTrendsLINKS
• CoalitionforEvidence-Based
Policy
• CommunitiesinSchools
• CrimeSolutions.gov
• CurriculumAssociates
• DepartmentofEducation’s
What Works Clearinghouse
• DepartmentofLabor’s
CLEAR
• EarlyIntervention
Foundation (UK)
• Evidence-basedPrevention
and Intervention Support
Center (EPISCenter)
• FunctionalFamilyTherapy
LLC
• FindYouthInfo.gov
• HanoverResearch
• HomeVisitingEvidenceof
Effectiveness (HomVEE)
• MultisystemicTherapy
Services
• OfficeofJuvenile
Justice and Delinquency Prevention’s (OJJDP) Model Programs Guide
• PeabodyResearchInstitute
• PromisingPracticesNetwork
• RegionalEducational
Laboratories
• SubstanceAbuseand
Mental Health Services Administration’s (SAMHSA) National Registry of Evidence-based Programs and Practices (NREPP)
• TeenPregnancyPrevention
• WhatWorksNetwork(UK)
• WashingtonStateInstitute
for Public Policy (WSIPP)
• America Achieves
• Centerfor Prevention of Youth Behavior Problems, University of Alabama
• US Department of Education
• Edna McConnell Clark Foundation
• Social Enterprise Initiative, Harvard Business School
• Madison Initiative,
Hewlett
Foundation
• Centerfor the Social
Organization
of Schools
• Venture
Philanthropy
Partners
• WTGrant
49
Appendix 2: Current efforts to develop and inform a common evidence framework
a. Recently revised Education Department General Administrative Regulations (EDGAR)—US Department of Education:
Recent Regulatory Amendment, August 13, 2013
http://www.gpo.gov/fdsys/pkg/FR-2013-08-13/pdf/2013-19390.pdf
b. Common Guidelines for Education Research and Development—US Department of Education Institute of Education Sciences and the National Science Foundation:
Common Guidelines for Research and Development: A Report from the Institute of Educations Sciences, US Department of Education, and the National Science Foundation, August 2013 http://ies.ed.gov/pdf/CommonGuidelines.pdf
c. CrossAgency Platform for Judging Evidence—in development by the
US Departments of Education (Institute of Education Sciences), Health and Human Services, and Labor (informal federal interagency work group):
Exploring a Cross-Agency Platform for Judging Evidence: Resources for Federal Agencies http://findyouthinfo.gov/docs/Common%20Evidence%20Framework- Draft_508_3-11-13.pdf
d. Framework for Continuous Evidence Building—in development by the Edna McConnell Clark Foundation, MDRC, and The Bridgespan Group:
Assessing an Organization’s Evidence of Effectiveness
http://www.emcf.org/fileadmin/media/PDFs/emcf_levelsofeffectiveness.pdf
e. Nesta Standards of Evidence—an approach used by the UK charity to measure the impact of a range of programs and investments, which began with a focus on impact investments:
Standards of Evidence: An Approach that Balances the Need for Evidence with Innovation http://www.nesta.org.uk/sites/default/files/standards_of_evidence.pdf
Standards of Evidence for Impact Investing
http://www.nesta.org.uk/sites/default/files/standards_of_evidence_for_ impact_investing.pdf
50
Appendix 3: Landscape of clearinghouses (page 1 of 5) USfocused clearinghouses
CLEARINGHOUSE NAME
WEBSITE
SPONSOR1
DOMAIN
EVIDENCE BASE
Abdul Latif Jameel Poverty Action Lab
http://www. povertyactionlab.org/
Non-government
Poverty
Policy lessons
Best Evidence Encyclopedia
http://www.best evidence.org/index.cfm
Government- funded
Education
Programs; Meta-analyses
Best Practices Registry for Suicide Prevention
http://www.sprc.org/bpr
Government- funded
Suicide
Programs
Blueprints for Healthy Youth Development
http://www.blueprints programs.com/
Non-government
Youth Development
Programs
California EvidenceBased Clearinghouse for Child Welfare
http://www.cebc4cw.org/
Government- funded
Child Welfare
Programs
California Healthy Kids Resource Center
http://www.california healthykids.org/index
Government- funded
Youth
Programs
Center on Know ledge Translation for Disability and Rehabilitation Research
http://www.ktdrr.org/
Government- funded
Disability and Rehabilitation
Systematic reviews
Child Trends LINKS (Lifecourse Interventions
to Nurture Kids Successfully)
http://www.childtrends. org/what-works/
Non-government
Children/Youth
Programs
Coalition for EvidenceBased Policy (CEBP) – Programs That Work
http://evidencebased programs.org/
Non-government
Social Programs
Interventions (programs and strategies)
CEBP – Top Tier Evidence
http://toptierevidence. org/
Non-government
Social Programs
Interventions (programs and strategies)
Collaborative for Academic, Social, and Emotional Learning Guide
http://www.casel.org/ guide
Non-government
Social and Emotional Learning
Programs
CrimeSolutions.gov
http://www.crime solutions.gov/
Government-run
Criminal Justice
Programs; Practices
Department of Education’s What Works Clearinghouse
http://ies.ed.gov/ncee/ wwc/
Government-run
Education
Interventions
51
(page 2 of 5)
CLEARINGHOUSE NAME
WEBSITE
SPONSOR1
DOMAIN
EVIDENCE BASE
Department of Labor’s Clearing house for Labor Evaluation and Research
http://clear.dol.gov/
Government-run
Labor
Programs; Systematic reviews
EvidenceBased Practices for Substance Use
http://lib.adai. washington.edu/ ebpsearch.htm
Government- funded
Substance Use
Interventions
FindYouthInfo.gov
http://www. findyouthinfo.gov/
Government-run
Youth
Programs
Healthy Communities Network database of Promising Practices
http://www.healthy communitiesinstitute.com/ healthy-communities- network-2/ ; e.g. http://www.healthier centraloregon.org/index. php?module=PromisePra ctice&controller=index& action=index
Non-government
Community Health
Practices
Home Visiting Evidence of Effectiveness
http://homvee.acf.hhs. gov/
Government-run
Home Visiting
Programs
My Brother's Keeper
http://mbk.ed.gov/
Government-run
Youth
still to launch
National Dropout Prevention Center/ Network
http://www.dropout prevention.org/
Non-government
Dropout Prevention
Programs; Strategies
National Guideline Clearinghouse
http://www.guideline. gov/index.aspx
Government-run
Health
Guideline
Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) Model Programs Guide
http://www.ojjdp.gov/ mpg/
Government-run
Juvenile Justice; Delinquency Prevention
Programs
OJJDP’s Strategic Planning Tool
https://www.national gangcenter.gov/SPT/
Government-run
Gangs
Programs
Promise Neighborhoods Research Consortium
http://promise neighborhoods.org/ index.html
Government- funded
Neighborhoods
Programs; Policies; Kernels
RAND’s Promising Practices Network
http://www.promising practices.net/
Non-government
Children & Families
Programs
Researchtested Intervention Programs
http://rtips.cancer.gov/ rtips/index.do
Government-run
Cancer
Programs
52
(page 3 of 5)
CLEARINGHOUSE NAME
WEBSITE
SPONSOR1
DOMAIN
EVIDENCE BASE
Substance Abuse and Mental
Health Services Administration’s National Registry of Evidencebased Programs and Practices
http://nrepp.samhsa.gov/
Government-run
Substance Abuse; Mental Health
Interventions
Strengthening Families Evidence Review
http://familyreview.acf. hhs.gov/Default.aspx
Government-run
Children & Families
Programs
Teen Pregnancy Prevention
http://www.hhs.gov/ash/ oah/oah-initiatives/teen_ pregnancy/index.html
Government-run
Teen Pregnancy Prevention
Program
The Campbell Collaboration
http://www.campbell collaboration.org/lib/
Non-government
Education; Criminal Justice; Social Welfare; International Development
Systematic Reviews
The Cochrane Library
http://www.thecochrane library.com/view/0/ index.html
Non-government
Healthcare/ Medicine
Systematic Reviews
The Community Guide
http://www.the communityguide.org/
Government- funded
Health
Systematic Reviews
United States Interagency Council on Homelessness' Solutions Database
http://usich.gov/usich_ resources/solutions/
Government-run
Homelessness
Solutions
Washington State Institute for Public Policy
http://www.wsipp. wa.gov/
Government-run
Public Policy
Practices
What Works for Health (Wisconsin)
http://whatworksfor health.wisc.edu/ background.php
Non-government
Community Health
Programs; Policies
What Works in Reentry Clearinghouse
http://whatworks. csgjusticecenter.org/
Government- funded
Reentry
Interventions
Source: Landscape is based on Bridgespan analysis through interviews and secondary research. 1. ‘Government-run’ clearinghouses are those primarily run and funded by federal, state, or local
governments; ‘Government-funded’ clearinghouses are those primarily funded by governments but run independently.
53
Example international clearinghouses (page 4 of 5)
CLEARINGHOUSE NAME
WEBSITE
SPONSOR
DOMAIN
EVIDENCE BASE
3ie/DFID Systematic Review database
http://www.3ieimpact.org/ en/evidence/systematic- reviews/
UK
Development
Systematic Reviews
Centre for Ageing Better
http://www.centrefor ageingbetter.com/
UK
TBD
(launch in 2015)
Centre for Excellence and Outcomes in Children and Young People's Services
http://www.c4eo.org.uk/ home.aspx
UK
Children, Youth & Families
Practices
Centre for Reviews and Dissemination database
http://www.crd.york.ac.uk/ crdweb/HomePage.asp
UK
Healthcare/ Medicine
Systematic Reviews (primarily)
Commissioning Toolkit – Parenting Programmes
https://www.education.gov. uk/commissioning-toolkit/ Programme/Index
UK
Parenting Programmes
Programmes/ Interventions
Early Intervention Foundation Guidebook
http://guidebook.eif.org.uk/
UK
Early Intervention
Programs
Education Endowment Foundation Toolkit
http://educationendowment foundation.org.uk/toolkit/
UK
Education
Topics
Evidence for
Policy and Practice Information and Co ordinating Centre
http://eppi.ioe.ac.uk/ cms/Default.aspx? tabid=53
UK
Social Research
Systematic reviews
Health Evidence
http://www.health evidence.org/
Canada
Public Health
Systematic reviews
Investing In Children
http://investinginchildren.eu/
EU
Children's Services
Interventions
National Institute for Health and Care Excellence Guidance
http://www.nice.org.uk/ guidance
UK
Health
Guideline
Project Oracle – Children & Youth Evidence Hub
http://project-oracle.com/
UK
Children & Youth
Projects
The National Board of Health and Welfare (Socialstyrelsen)’s MetodGuiden
http://www.social styrelsen.se/evidens baseradpraktik
Sweden
Social Work
Interventions
54
(page 5 of 5)
CLEARINGHOUSE NAME
WEBSITE
SPONSOR
DOMAIN
EVIDENCE BASE
What Works Centre for Crime Reduction
http://www.college.police. uk/en/20018.htm
UK
Crime Reduction
Briefing (summarizes systematic review)
What Works Centre for Local Economic Growth
http://whatworks growth.org/
UK
Economic Growth
Evidence reviews
Source: Landscape is based on Bridgespan analysis through interviews and secondary research.
55
Appendix 4: Web analytics for information sources (page 1 of 3) Relative Estimates of Unique Monthly Visitors for Information Sources
(Average from 10/13–9/14)
Clearinghouse
Other information source
20K –
0–
1.5M –
0–
1. Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-based Programs and Practices
2. American Association of School Administrators
3. Council of Chief State School Officers
4. California Evidence-Based Clearinghouse for Child Welfare
5. Coalition for Evidence-Based Policy
6. Collaborative for Academic, Social, and Emotional Learning
Source: Compete.com metrics for unique monthly visitors; Bridgespan analysis
Note: Based on experience with Compete.com, metrics are likely underestimated and should be used as relative references.
56
Teachers Pay Teachers Edutopia Eric.ed.gov National Guideline Clearinghouse
National Governers Association SAMHSA’s NREPP1 Wallace Foundation AASA2 Child Trends CrimeSolutions.gov CCSSO3 Child Welfare League of America The Community Guide The New Teacher Project Casey Family Program FindYouthInfo.gov CEBC4 CEBP5 – Top Tier Evidence Campbell Collaboration CASEL6 Chapin Hall RAND’s Promising Practices Network Blueprints The Cochrane Library CEBP – Programs That Work Research-tested Intervention Programs California Healthy Kids Promise Neighborhoods Healthy North Texas
Web Analytics Provided by Clearinghouses
(various metrics and timeframes) (page 2 of 3)
CLEARINGHOUSE
VISITORS
VISITS
PAGE VIEWS
OTHER
Blueprints for Healthy Youth Development
51,000 visitors over 6 months; 8,500 avg. visitors per month, or
280 per day
Not received
• 169,600 total page views over 6 months; 28,300 avg. total page views per month, or 940 per day
• 116,700 unique page views over 6 months; 19,400 avg. unique page views per month, or 650 per day
• 2:22 avg. time spent on a page
• New visitors: 2.91 pages per visit and 146.76 avg. duration (sec.)
• Returning visitors: 4.38 pages per visit and 303.14 avg. duration (sec.)
California EvidenceBased Clearinghouse for Child Welfare
63,500 visitors over 3 months; 21,200 avg. visitors per month or
700 per day
Not received
• 211,193 page views over 3 months; 70,400 avg. page views per month or 2,350 per day
• 2.73 pages per site visit
• 2:08 min avg. per site visit
Child Trends LINKS
503,900 visitors over 10 months to ChildTrends. org; 50,400 avg. visitors per month or 1,680 per day
Not received
• 12,200 page views over 10 months
of What Works
List of Programs; 1,200 avg. page views per month or 40 per day
• 54,200 page views over 10 months
of What Works Programs; 5,400 avg. page views per months or
180 per day
• An example LINKS Synthesis report received 600 views over
10 months, or
60 avg. per month
Coalition for EvidenceBased Policy
~400 visitors per day for ‘Top Tier Programs’
Not received
Not received
Not received
CrimeSolutions.gov
289,900 unique visitors over
12 months; 24,200 avg. unique visitors per month or 790 per day
368,000 visits over
12 months; 30,700 avg. visits per month or 1,010 per day
• 742,900 page views over 12 months; 61,900 avg. page views per month or 2,040 per day
• 2.02 pages per session
• 2:25 min avg. session duration
Department of Education’s What Works Clearinghouse
~8,000 visits per day to www.ies.ed.gov/ ncee/wwc
Not received
Not received
• 235,000 avg. monthly hits over 18 months
• 627,000 total downloads for all product types over 12 months
57
(page 3 of 3)
CLEARINGHOUSE
VISITORS
VISITS
PAGE VIEWS
OTHER
Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide
~200-350 visitors per day or ~800 visitors per day if there is a big email blast (from interview)
99,600 total visits over
9 months; 11,100 avg. total visits per month or 370 per day (from report)
Not received
Not received
Substance Abuse and Mental Health Administration’s National Registry of Evidencebased Practices and Programs
30,500 unique visitors per month or 1,020 per day (as of Aug 2013)
105,000 total visits over
2 months; 52,500 avg. total visits per month and 1,750 per day (as of Feb 2014)
• 389,000 page views over 2 months; 195,000 avg. page views per month and 6,500 per day (as of Feb 2014)
• Top most viewed intervention summary received 3,900 page views over 2 months
• Top most viewed systematic review page received 3,200 page views over 2 months
Washington State Institute for Public Policy
Not received
15,700 visits over 6 months; 2,600 avg. visits per month or 90 per day
• 31,600 total page views over 6 months; 5,300 avg. total page views per month or 180 per day
• 25,800 unique page views over 6 months; 4,300 avg. unique page views per month or 140 per day
• 2:26 min avg. visit duration
• 4,900 total downloads over 6 months; 820 avg. average downloads per month or 26 per day
Note: Data received from differently clearinghouses may be from a variety of different web analytics tools and may represent slightly different metrics, and therefore should be used for directional purposes only. Numbers have also been rounded slightly.
Source: From interviews with or reports provided directly by the clearinghouses
58
Appendix 5: US-focused clearinghouses by domain (page 1 of 3)
Legend
Primary outcome focus of clearinghouse
Secondary outcome focus of clearinghouse
Outcome addressed by a few interventions, due to primary focus of clearinghouse
CLEARINGHOUSE NAME
YOUTHSPECIFIC OUTCOMES
NON AGESPECIFIC OUTCOMES
DOMAIN
Abdul Latif Jameel Poverty Action Lab
Best Evidence Encyclopedia
Poverty
Education
Best Practices Registry for Suicide Prevention
Suicide
Blueprints for Healthy Youth Development
Youth Development
California Evidence Based Clearinghouse for Child Welfare
Child Welfare
California Healthy Kids Resource Center
Youth
Center on Knowledge Translation for Disability and Rehabilitation Research
Disability and Rehabilitation
Child Trends LINKS (Lifecourse Inter ventions to Nurture Kids Successfully)
Children/Youth
Coalition for Evidence Based Policy (CEBP) – Programs That Work
CEBP – Top Tier Evidence
Social Programs
Social Programs
Collaborative for Academic, Social, and Emotional Learning Guide
Social and Emotional Learning
CrimeSolutions.gov
Criminal Justice
59
Education
Social & Emotional Health
Child Welfare
Juvenile Justice/Problem Behavior Youth Mental Health & Substance Abuse Youth Public Health
Teen Pregnancy
Children & Youth (OVERALL)
Family Relationships
Labor & Workforce Development Criminal Justice & Reentry
Mental Health & Substance Abuse Public Health
Healthcare/Medicine
Welfare
Disabilities
Elderly
Housing/Homelessness
Local Economy
Community Health (OVERALL) International Development
Legend
Primary outcome focus of clearinghouse
Secondary outcome focus of clearinghouse
Outcome addressed by a few interventions, due to primary focus of clearinghouse
(page 2 of 3)
CLEARINGHOUSE NAME
YOUTHSPECIFIC OUTCOMES
NON AGESPECIFIC OUTCOMES
DOMAIN
Department of Education’s What Works Clearinghouse
Education
Department of Labor’s Clearinghouse for Labor Evaluation and Research
EvidenceBased Practices for Substance Use
Labor
Substance Use
FindYouthInfo.gov
Youth
Healthy Communities Network database of Promising Practices
National Guideline Clearinghouse
Community Health
Home Visiting Evidence of Effectiveness
Home Visiting
My Brother’s Keeper
Youth – specific domains TBD
National Dropout Prevention Center/ Network
Dropout Prevention
Health
Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) Model Programs Guide
Juvenile Justice; Delinquency Prevention
OJJDP’s Strategic Planning Tool
Gangs
Promise Neighborhoods Research Consortium
Neighborhoods
RAND’s Promising Practices Network
Children & Families
Researchtested Intervention Programs
Cancer
60
Education
Social & Emotional Health
Child Welfare
Juvenile Justice/Problem Behavior Youth Mental Health & Substance Abuse Youth Public Health
Teen Pregnancy
Children & Youth (OVERALL)
Family Relationships
Labor & Workforce Development Criminal Justice & Reentry
Mental Health & Substance Abuse Public Health
Healthcare/Medicine
Welfare
Disabilities
Elderly
Housing/Homelessness
Local Economy
Community Health (OVERALL) International Development
Legend
Primary outcome focus of clearinghouse
Secondary outcome focus of clearinghouse
Outcome addressed by a few interventions, due to primary focus of clearinghouse
(page 3 of 3)
YOUTHSPECIFIC OUTCOMES
NON AGESPECIFIC OUTCOMES
CLEARINGHOUSE NAME
DOMAIN
Substance Abuse
and Mental Health Administration’s National Registry
of Evidencebased Practices and Programs
Substance Abuse; Mental Health
Strengthening Families Evidence Review
Children & Families
Teen Pregnancy Prevention
Teen Pregnancy Prevention
The Campbell Collaboration1
The Cochrane Library
Education; Criminal Justice; Social Welfare; Int’l Dev’t
Healthcare/ Medicine
The Community Guide
Health
United States Interagency Council on Homelessness’ Solutions Database
Homelessness
Washington State Institute for Public Policy
What Works for Health (Wisconsin)
Public Policy
Community Health
What Works in Reentry Clearinghouse
Reentry
Source: Outcome coverage was approximated through Bridgespan analysis and based on interviews and secondary research. For broader websites, outcome coverage was assessed primarily based on the clearinghouse functionality (e.g. FindYouthInfo.gov).
1. Outcomes covered by ‘Social Welfare’ for The Campbell Collaboration was based on:
http://www.campbellcollaboration.org/SWCG_Our_Group/area_of_work.php.
2. Public and Youth Health are distinguished from Healthcare/Medicine as interventions implemented by a community organization rather than clinical interventions implemented by an institution.
61
Education
Social & Emotional Health
Child Welfare
Juvenile Justice/Problem Behavior Youth Mental Health & Substance Abuse Youth Public Health
Teen Pregnancy
Children & Youth (OVERALL)
Family Relationships
Labor & Workforce Development Criminal Justice & Reentry
Mental Health & Substance Abuse Public Health
Healthcare/Medicine
Welfare
Disabilities
Elderly
Housing/Homelessness
Local Economy
Community Health (OVERALL) International Development
Appendix 6: Comprehensiveness of clearinghouses (page 1 of 2)
CLEARINGHOUSE NAME
INTER VENTIONS IN REVIEW QUEUE
INTER VENTIONS LACKING MINIMUM STANDARDS/ ELIGIBILITY
INTER VENTIONS REVIEWED BUT NOT RATED
INTER VENTIONS LACKING STRONG EVIDENCE1
INTER VENTIONS LACKING EFFECTIVE NESS/ IMPACT
Blueprints for Healthy Youth Development
California Evidence Based Clearinghouse for Child Welfare
✔
✔
✔
Child Trends LINKS
✔
✔
Coalition for Evidence Based Policy (CEBP) – Programs That Work
✔
2
CEBP – Top Tier Evidence
✔3
2
CrimeSolutions.gov
✔4
✔4
✔
Department of Education’s What Works Clearinghouse
✔6
✔
✔
✔
✔
Department of Labor’s Clearinghouse for Labor Evaluation
and Research
✔
✔
✔
FindYouthInfo.gov5
✔
✔
Home Visiting Evidence of Effectiveness
✔
✔
✔
✔
Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide
✔
RAND’s Promising Practices Network
✔
Substance Abuse
and Mental Health Administration’s National Registry
of Evidencebased Practices and Programs
✔6
✔
7
Teen Pregnancy Prevention
✔
✔
✔
✔
Washington State Institute for Public Policy
✔
✔
Source: Assessment of coverage is based on Bridgespan analysis through interviews and secondary research.
62
(page 2 of 2)
Note: Clearinghouses present these types of information either at the intervention or study level; in particular, information is often provided at the study level for ‘Interventions lacking minimum standards/eligibility’ and ‘Interventions reviewed but not rated.’
1. Designation of ‘strong evidence’ varies by clearinghouse; however, unless clearinghouses include
programs that show low/little/no evidence, they do not receive a check in this category.
2. While the Coalition for Evidence-Based Policy does rate programs as ‘Near Top Tier’, and includes
not rated programs on the ‘Programs That Work’ site, all listed programs are determined to have
strong evidence to a certain extent and are promising with regards to effectiveness.
3. On its ‘Top Tier Evidence’ site, the Coalition for Evidence-Based Policy lists but does not provide
additional information on interventions reviewed but not rated to date.
4. CrimeSolutions.gov’s ‘Insufficient evidence list’ is provided in a document that needs to be
downloaded from the website, and encompasses interventions with a lack of strong evidence.
5. FindYouthInfo.gov leverages databases from CrimeSolutions.gov and Teen Pregnancy
Prevention; assessments are based on information available across both databases.
6. National Registry of Evidence-based Programs and Practices and What Works Clearinghouse
include interventions in progress of being reviewed, but not those still to be reviewed.
7. National Registry of Evidence-based Programs and Practices does not assess an intervention’s effectiveness.
63
Appendix 7: Intervention-specific information captured systematically on clearinghouses (page 1 of 2)
CLEARINGHOUSE NAME
COSTS
COST/ BENEFIT COMPARISON
POPULATION CHARACTER ISTICS (MORE THAN AGE)
IMPLE MENTATION DETAILS
CONTACT INFORM ATION
Blueprints for Healthy Youth Development
✔
✔
✔
✔
✔
California Evidence Based Clearinghouse for Child Welfare
✔
✔
Child Trends LINKS
✔
✔
Coalition for Evidence Based Policy (CEBP) – Programs That Work
✔
CEBP – Top Tier Evidence
✔
CrimeSolutions.gov
✔
✔3
✔
✔
✔
Department of Education’s What Works Clearinghouse
✔4
✔4
Department of Labor’s Clearinghouse for Labor Evaluation and Research
FindYouthInfo.gov5
✔
✔
Home Visiting Evidence of Effectiveness
✔
✔
✔
Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide
✔
✔3
✔
✔
✔
RAND’s Promising Practices Network
✔
✔
✔
Substance Abuse
and Mental Health Administration’s National Registry
of Evidencebased Practices and Programs
✔
✔
✔
✔
Teen Pregnancy Prevention
✔
✔
Washington State Institute for Public Policy
✔
✔
Source: Assessment of inclusion is based on Bridgespan analysis through interviews and secondary research.
Note: Assessment is intended to capture types of intervention-specific information that are captured
64
(page 2 of 2)
systematically, extracted, and displayed as an identified section in an intervention profile; information is not designated as available if it is only captured on an ad-hoc basis, within a program description, etc.
1. CrimeSolutions.gov and OJJDP’s Model Programs Guide use the same underlying database.
2. FindYouthInfo.gov leverages databases from CrimeSolutions.gov and Teen Pregnancy Prevention;
assessments are based information available across both databases.
3. CrimeSolutions.gov and OJJDP’s Model Programs Guide do not have sections labeled as
cost/benefit, but this is sometimes included within cost information.
4. What Works Clearinghouse captures cost and contact information in full intervention reports that can be downloaded from the website, but not directly on the intervention profile page.
65
BOSTON
2 Copley Place, 7th Floor Suite 3700B
Boston, MA 02116
Tel: 617 572-2833
NEW YORK
112 West 34th St., Ste. 1510 New York, NY 10120
Tel: 646 562-8900
SAN FRANCISCO
465 California St., 11th Fl. San Francisco, CA 94104 Tel: 415 627-1100
This work is licensed under the Creative Commons BY-NC-ND License.
To view a copy of this license, visit www.bridgespan.org/terms-of-use.aspx
www.bridgespan.org
 |