Systems Approach for Better Education Results (SABER)

Frequently Asked Questions What is SABER’s overall approach? What is the methodology behind SABER? How do I use SABER’s frameworks and analyses? How can I participate in SABER? What is SABER’s overall approach? What is SABER? The Systems Approach for Better Education Results (SABER) is an initiative to collect and disseminate comparative data and knowledge on education policies, to help countries systematically evaluate and strengthen their education systems. The World Bank Group is working with partners around the world – not only to collect the information – but also to develop and apply diagnostic tools that evaluate the quality of education policies and benchmark them against evidence-based global standards. SABER fills a gap in the availability of policy data, information, and knowledge on what matters most to improve the quality of education and help countries progress toward learning for all.What is SABER’s focus? Education outcomes—such as enrollment, completion, and student learning—depend on many factors. An important one is the quality of the policies and institutions governing the system, but until now there has been too little systematic, detailed information on them. SABER’s main focus is to collect data on these policies and institutions and to analyze their quality. What is a SABER “domain”? In SABER, the detailed data-gathering and analysis are organized by “domains”, or areas of education policymaking within an education system. These include levels of education (Early childhood education and development, Primary and secondary schooling, Workforce Development, and Tertiary Education), types of quality resources and supports (Finance, School Health and School Feeding, and Teachers), areas of governance (engaging the private sector and school autonomy and accountability), information sources (education management information systems and student assessment), and complementary inputs and cross-cutting themes (equity and inclusion, information communication technologies, and resilience). What does the “systems approach” mean? SABER is based on the systems approach to education analysis and reform, which is at the heart of the World Bank Group Education Strategy 2020: Learning for All. The systems approach recognizes that improving education requires much more than just increasing resources – it requires strengthening all the factors that improve learning for all children and youth. This means making sure that the system’s policies for governance, accountability mechanisms, information systems, financing rules, and school management are all aligned toward achieving learning for all. SABER embodies this systemic definition of what types of policies matter, both within each domain and across domains. Systems thinking can shed light on how parts of a system can be optimized, including the results chain and related feedback loops that enable the system to achieve desired results. Do the SABER analyses and ratings implicitly assume that there is only one correct way to organize an education system? No. There is a great diversity of approaches to education and there are different routes to educational success. In the “What Matters” papers, SABER domain teams—in consultation with leading experts in the field—try to identify some common policy and institutional threads that run through most of the more successful experiences. They then use those broadly common features to guide the analysis and benchmarking exercises, but not in a way that requires a specific narrow policy or institutional form as a condition for a good rating. Users can always draw on the underlying SABER data on policies and institutions to devise their own analyses and ratings, if they prefer. Why is SABER focusing on measuring the quality of formal policies and institutions (policy intent) as well as policy implementation? The SABER initiative is continuing to make a major push over the medium term to collect data on the quality of policies and institutions in education. This is an area on which there is currently very little systematic, comparable data, and one on which it is possible to make major strides in a short time. Policy implementation is crucial too – the best-designed education policy means little if it is not implemented effectively. This is why SABER’s core data-gathering work incorporates not only policies, but also the institutional frameworks that are essential for implementing those policies, as well as implementation indicators that can be captured reliably at the system level. Under the SABER umbrella, the domain teams are beginning to test and utilize new tools for evaluating how well policies are implemented. These tools will be available for others to apply, and are currently being applied where funding and opportunities permit. Additionally, efforts are being made to explore the links between SABER policy intent findings and policy implementation, service delivery, and/or learning data. Is SABER a blueprint for a top-down, government-managed approach to education? Not at all. SABER’s vision is to empower all education stakeholders with information they need to make decisions. For education systems to promote learning for all, they need to set and appropriate policy and institutional framework and to implement it effectively. These are government responsibilities but the framework also encompasses the whole education system, including private schools and teachers and autonomously managed schools. SABER includes domains aimed at ensuring that systems release bottom-up energy and creativity, as well as providing the right supervisory and management framework for that to happen. Where’s the SABER ranking of countries? SABER does not rank countries’ education systems or even domain subsystems. Instead, it provides detailed data, as well as analyses and benchmarks, to help countries do the hard work of improving their systems. Other rankings, such as PISA and TIMSS rankings of student learning—are motivating countries to turn to SABER for information on how to make improvements. But SABER focuses on providing detailed guidance in particular policy areas, rather than ranking. What is the methodology behind SABER? Who are the experts who came up with these judgments? The team for each domain is led by an experienced senior expert in the field from the World Bank Group, working with a dedicated team of colleagues. Each domain team works with a small advisory group of outside experts—both academics and practitioners—to help develop the “What Matters” paper, the indicators, and the rubric. The paper and rubric undergo a formal peer review by researchers and operational staff, with management oversight and involvement, before they are rolled out. Typically, many other interested parties are consulted as well during the process of developing and piloting the approaches. Are the ratings comparable across different policy domains? That is, do “Emerging” and “Advanced” imply the same level of development in Teacher Policy as in Assessment Policy? Yes. The World Bank Group has tried to make the ratings roughly comparable across domains through the methodology used to set the rubrics used in coming up with the ratings. Domain teams used the actual distribution of policies in pilot countries to create the rubrics, so that “Latent” summarizes the type of policies found in the least developed and lowest-performing education systems, while “Advanced” represents the type of policies found in the highest-performing systems. The domains use very different indicators and no strict comparability is possible across domains. But the similarity in methodology across domains should be sufficient to allow countries to use the ratings to identify areas of relative strength and weakness (relative to the range of quality observed across countries) in their policy framework. How are SABER data collected and analyzed?The SABER policy data-gathering and evaluations are designed to be carried out relatively quickly and cost-effectively. Since the goal is to analyze the quality of the policy and institutional framework, rather than do a detailed assessment of school-level conditions, most of the data-gathering can usually be done centrally by knowledgeable experts, at either the national or provincial level. While there is some variation across domains, the typical model for data collection and analysis is as follows: “What Matters” paper:The “What Matters” paper surveys the best evidence and experience in the domain and uses that survey to identify the elements of the policy and institutional framework that matter most for improving education outcomes, including learning. Indicators and scoring rubric: The “What Matters” paper is used to identify indicators of policy and institutional development, as well as the data source for each indicator. A rubric is then developed for combining the indicators to come up with ratings of the country’s progress within each domain. These indicators and scoring rubrics will generally be objective, rather than subjective, to ensure cross-country comparability and replicability. Data-collection instrument: An instrument for collecting data for the policy and institutional indicators is developed. This is not an instrument for a large-sample survey, but essentially a survey for one respondent—an experienced principal investigator in the country—to fill out using information from key informants, documents, and other sources. Data collection: In the typical model, an experienced principal investigator will collect the policy information and data necessary to fill out the data-collection instrument, by drawing on his or her knowledge of the system and on government contacts. Data collection can usually be completed within a few weeks in this way. An alternative approach used in some domains is to convene a workshop of experts, including government officials, and use that group process to collect the evidence and code data. In either case, data sources are clearly identified and made public when the data are posted. Analysis: The domain team uses the data to analyze how developed the country’s education policies and institutions are, from the perspective of achieving key education goals. In the process, it will also generate benchmarks of progress in those specific areas against other countries or provinces. In some domains, these evaluations will be embedded in a more in-depth report discussing policy options and relevant experiences from other countries. While the principal investigator in each country may carry out the initial analysis, the central SABER team for each domain is responsible for completing the analysis and ensuring cross-country comparability. Validation and discussion: The domain team presents the data to World Bank regional team leaders for validation with government officials, to ensure that SABER reports the country’s policies and institutions correctly. Any corrections to the information on which the analysis is based will be incorporated before publication. In addition, the team or, preferably, the World Bank Group regional staff member) will discuss the resulting report with government counterparts. Finally, the report is reviewed and approved by the World Bank’s education sector board before it is finalized and made public. Publication of analyses and data: Both the country report and (in the near future) the data underlying it are made public on the SABER website. This ensures that users can benefit from expert analyses, if they choose to, but that they can also carry out their own analyses and benchmarking using the SABER data. How does SABER data illuminate policy choices?Analytical: Even before data collection takes place, SABER highlights key policy choices for decision makers. The SABER data collection and analysis in each domain are built around an analytical framework that highlights for policymakers and other key actors the most important (and actionable) policy choices to spur learning. The analytical framework is based on a survey of the best global evidence, including results from rigorous impact evaluations, other research, and professional standards. The framework can be used to structure conversations about system development and potential reforms, drawing on the wealth of information that policymakers and others already have about the education system.Descriptive: SABER systematically collects and catalogues data and information on the policy and institutional choices that countries have made in each education domain.The choice of data to collect is based on thorough background research aimed at identifying the policies and institutions that are most likely to be related systematically to student learning. These data will allow policymakers or concerned citizens in any country to see, at a glance, what their official policies and institutions are in the area of teacher policy. This ability to make policy choices more transparent, in a detailed way, should be a major contribution of SABER. Very few people (even within an education ministry) know all of their country’s policies in a given domain, let alone across all domains; and far fewer will know other countries’ policies that well. SABER will empower them by making the invisible parts of the education sector more visible.Evaluative: SABER uses data to evaluate the level of development of education policies and institutions. In each domain, the team uses extensive literature surveys and practitioner input to identify policy and institutional choices most often seen in the best-performing and fastest-improving education systems. The team codifies these good-practice choices in a set of indicators and then uses the data collected for those indicators to evaluate how advanced each education system in the given area. SABER has been standardized so that progress along each dimension is summarized in a four-point scale, ranging from Latent (the lowest level) through Emerging and Established to Advanced (the highest level), based on international benchmarks. To make concrete what these levels signify, consider the following example from SABER-Student Assessment. How do I use SABER’s frameworks and analyses? Why doesn’t the “What Matters” paper in each domain provide more detailed how-to instructions? The “What Matters” papers will not be able to serve as a “how-to” manual for education policymakers but should nonetheless be useful as a stand-alone document for many purposes—for example, to guide the policy dialogue in a domain, to provide the rationale for the SABER tool, and to point the way toward other resources that can provide more detailed guidance where necessary. What if I disagree with the team’s judgments about what the best policies and institutions are in this area of education? Contact the team with your concerns by submitting your comments and questions. The indicators and rubrics for each domain are expected to evolve over the coming months and years, as more information becomes available, and the SABER team welcomes further evidence and opinions on what matters and how to measure it. Even if you disagree with one or more of these judgments about “what matters”, do use the SABER database of policies of institutions to learn from many countries’ experiences and draw your own conclusions. It is not just the benchmarking and ratings, but also the data itself that can be a tool for empowerment.What if I don’t think the SABER data correctly reflect the policy framework in my country? Submit corrections or updates by using the SABER “Contact Us” page. If you want to check first on where the posted data come from, the relevant policies, regulations, and other sources of data will be posted on the “Laws and Policy Documents” page of the relevant SABER domain website. Any corrections to posted data will be incorporated during regular updates of the database. Any changes in the policy framework that come in after the publication of the country report will be listed and described on the relevant country page.When will the latest education reforms be reflected in the ratings in the domain I am interested in? Information on relevant new reforms to education policies and institutions will be posted to the relevant country’s data page. These updates will allow users to know how the policy framework has changed since the last round of data collection and ratings in the country. The domain ratings will not be updated immediately as each new piece of information is received because ratings depend on a variety of indicators and the SABER team will not be able to update all of those indicators instantaneously. However, ratings will be updated each time the team is able to carry out a new round of data collection in the country/domain. Does SABER provide data on education systems only in low- and middle-income economies? No, it includes wealthier economies as well. In order to identify indicators of policies conducive to learning and to benchmark the ratings across countries, each domain includes policy data from the education systems that generate the highest levels of learning. Some, though not all, of these systems are in the most advanced economies. After the initial piloting and benchmarking phase, the World Bank Group will usually be able to provide funding for applications of SABER only in developing economies, but we are happy to work with those who want to apply the SABER framework and tools in other developed economies. How do countries participate in SABER? I want my country to participate in SABER. How can I make that happen? Please contact the central SABER team by filling in the “Contact Us” form. If you are a government official, you can also discuss your interest in SABER with the World Bank Group education manager or team leader working with your country. When is the best time for countries to engage with SABER? Since SABER provides actionable, comparable policy and institutional information about an education system that otherwise would be hard to obtain, any time can be an appropriate time to apply the SABER instruments. The analyses are likely to have their greatest effect if they are carried out just before windows of opportunity for action open up—for example, as part of the preparation of a major World Bank Group operation, or just before a new government takes office. How much will the SABER data-gathering and analysis cost for the country I’m working in? The cost of applying SABER varies by country and domain. Since the fixed costs of developing the framework and instruments have been financed by the World Bank Group and by other partners, each country team needs to pay only the marginal cost of gathering the data and preparing a brief analysis and benchmarking report. The cost for this core product line is typically much less than that of a typical World Bank Group sector study, even if several SABER domains are applied at once. It may be possible to expand the standard SABER application and analysis into a much more in-depth subsector study, as has been done in some countries; in this case, the costs will be higher, but of course so will the benefits.