Monitoring & Evaluation (M&E) Capacity - JDC Israel

15 JDC-Israel GENERAL M&E UPDATE Over the past year, and with generous support from the Harry and Jeanette Weinberg Foundation, JDCIsrael launched a major effort to upgrade its Monitoring & Evaluation (M&E) capacity, both within each of its five divisions and across a number of strategic areas. This effort has been guided by a steering committee led by Professors Gordon Berlin (MDRC) and David Weisburd (George Mason and The Hebrew University) and Dr. Aviva Zeltzer-Zubida. This section describes the work done across JDC-Israel as well as within each of its five divisions: Ashalim: Children and Families at Risk ELKA: JDC Institute for Leadership and Governance ESHEL: Elderly Israel Unlimited: People with Disabilities TEVET: Workforce Integration 16 The JDC-Israel Measurement & Evaluation Task Force JDC-Israel established an M&E Task Force at the beginning of 2016, building on the M&E point persons designated by each division three years ago as part of JDC’s global efforts to advance M&E throughout the organization. The Task Force includes the divisional point persons, as well as a representative from the MyersJDC-Brookdale Institute (MJB), which plays a critical role in the development of tools and guidelines. It meets for several hours each month. The developments described below are the result of the Task Force’s focused efforts. DESIGN Identify a social challenge Learn the issue Map best practices in Israel and abroad Design and plan the intervention NURTURE Launch a pilot Test and evaluate Refine model Build capacity of professionals and infrastructure ACCELERATE Assist government agencies and other partners to scale up and replicate nationally Provide oversight, ongoing consultation, professional training, and program development Standardized DNA Criteria for All JDC-Israel Programs JDC-Israel created the DNA (Design, Nurture, and Accelerate) model a few years ago to describe its program development strategy. The M&E Task Force was tasked with taking the model to the next level so it could be used as a tool to conceptualize, plan, manage, and monitor the different stages of program development across JDCIsrael. But first, the different stages needed to be defined more precisely and standardized criteria developed to gauge advancement from one stage to the next. Together with MJB, the task force created a tool that determines: 1. A program’s current stage of development 2. Clear criteria for when, if, and how it should advance to the next stage. Task Force members have now begun integrating this tool into the work processes of their divisions. In Eshel, Ashalim, and Tevet, detailed definitions of the steps required to implement each stage of the DNA model were developed and the new workflows are now being disseminated across the divisions. The divisions have begun training their staff on the DNA model, and a cross-JDC Israel seminar was held in 2016. 17 Evaluation Policy The M&E Task Force drafted a JDC-Israel Evaluation Policy putting the global M&E principles introduced by JDC in 2012 into the local context. The policy has been approved by the JDC-Israel Management Team and includes: Goals and aims of M&E in JDC-Israel programs Basic standards for all evaluation studies of JDC-Israel programs Specific M&E objectives that need to be accomplished at each stage of the DNA process Organizational capacities needed to support M&E in JDC-Israel Strategic partnership on M&E with Myers-JDCBrookdale Institute, and clarification of its role vis-a-vis external evaluation The evaluation policy lays out a number of strategic directions that will shape M&E work in the upcoming years: 1. Greater emphasis on evaluation of outcomes– both at the level of individual programs, but more importantly at the level of an area, a division, and across JDC-Israel. This includes defining shared outcome goals as well as shared measures. 2. Developing data systems for ongoing outcome measurement 3. Promoting more rigorous evaluation studies 4. Investing in long-term outcome studies in order to measure impact after programs have been handed over to the government A number of steps in these directions have been taken by the different divisions: SHARED OUTCOMES AND MEASURES The M&E task force is in the process of developing key JDC Israel-wide shared goals and outcome measures that will serve as the basis for evaluating all JDC-Israel programs. As JDC-Israel’s mission is to improve the system of care for Israel’s most vulnerable populations, these outcome measures relate both to the vulnerable populations and to the necessary changes in the system of care. These JDC-Israel outcomes serve as the foundation for developing a set of shared measures for programs for children and families (Ashalim), for elderly (Eshel), for people with disabilities (Israel Unlimited), and for programs that promote integration into the workforce (Tevet). In Ashalim, a “program success team” was created to identify key crosscutting program outcomes and measures for children and youth, families, and the community. The team prioritized measurements that are already in use by Ashalim partner ministries and/or were validated and are well known in the international literature. Both Eshel and Israel Unlimited have launched a similar process to define crosscutting program outcomes and develop shared measures in collaboration with MJB. Tevet, working with MJB, has taken a major step forward by launching a process to develop shared measures for employment together with all relevant government agencies. These shared measures would serve as the standard for evaluation of all government-funded employment programs in Israel. In 2016, Tevet, MJB, and the Ministries of the Economy and Social Affairs reached an agreement on measures that relate to key employment outcomes such as integration into the work force, job placement, and job retention. 18 2017 will serve as a pilot period for experimenting with the integration of these measures into the work of the different partners. The inter-organizational working group will follow the implementation of these measures with the aim of deciding on their permanent use. The group will also begin developing shared measures next year for additional employment outcomes, such as job advancement, soft-skills, and employability. Within Tevet, the shared measures are being integrated into the ongoing measurement system and system reports are being adjusted accordingly. The shared measures will also be used in all forthcoming evaluations. ONGOING OUTCOME MEASUREMENT Tevet utilizes an ongoing outcome measurement system, called Spot, in all its pilot programs. Many programs continue to use that system even after they have been transferred to the government for full-scale implementation. In the past year, Tevet began reviewing the requirements of the Spot system with the aim of developing a more updated model. Ashalim will pilot the data system developed by the National Program for Children and Youth at Risk in five Ashalim programs. If this proves successful, and based on the lessons learned, it plans to integrate this data system into most Ashalim programs in the coming years. As part of the global JDC initiative, similar systems are currently in development in all other divisions. These systems will integrate all relevant program materials, serve as the exclusive database for programs, and support data-informed decision-making processes and ongoing program monitoring. PROMOTING RIGOROUS EVALUATION IN JDC-ISRAEL To improve the sophistication and rigor of evaluation studies conducted on behalf of JDC-Israel, each division is seeking to expand the use of control groups and experimental research designs, including randomized control trials (RCT), where appropriate. A number of RCTs are now in active planning in Tevet, Eshel, and Ashalim and will be implemented starting next year. While some JDC programs have too many changing variables or other factors hindering the ability to implement experimental research designs, it is our intention to continue to promote rigorous evaluation studies in order to best support learning and decisionmaking processes. CROSSCUTTING AND LONG-TERM IMPACT STUDIES The Institute for Leadership and Governance has decided to initiate follow-up studies that examine longer-term outcomes of capacity building programs as well as holistic studies that examine the impact of a portfolio of interventions that are meant to move the needle on a shared challenge. Tevet has launched two major efforts in collaboration with MJB: an evaluation of the integration and expansion of Tevet programs by the government after the completion of their pilot stage, and an assessment of long-term employment outcomes for Tevet graduates based on data from the Central Bureau of Statistics. 19 106 external evaluations Close to 30% used some sort of comparison group Breakdown by division Breakdown by type 27% Eshel 23% Tevet 20% Ashalim 18% IU 18% ELKA 20% process 70% process & outcome 10% outcome The following pages highlight program findings from flagship programs in each of the five JDC-Israel divisions. PROGRAM FINDINGS All of JDC-Israel’s programs are evaluated at some point in their life cycle. In the past 6 years there have been: 20 Ashalim: Children and Families at Risk Better Together Better Together is a neighborhood change program that promotes the creation of effective communitybased systems in which professionals and residents work together to improve the wellbeing of children and youth in Israel’s disadvantaged communities and reduce their exposure to risk situations. Since 2006, the program has been implemented in some 43 communities with a broad range of ethnic backgrounds. Ashalim serves as the backbone organization in these communities and promotes an organized process of networking and learning. In the belief that it does indeed take a village to raise a child, Better Together emphasizes three levers of change: Voice and involvement of the residents— from a passive to a proactive community Integrated multi-disciplinary and multiorganizational mechanisms—from silos to coordination and cooperation among key communal organizations and their leaders. This facilitates a comprehensive approach to addressing challenges. Responses for children and youth—from fragmented services to a service continuum Program interventions and activities are chosen to fit the specific needs of each neighborhood. They include one-time special events to increase awareness of the program and mobilize resident support and engagement, and ongoing initiatives such as after school learning centers, leisure activities, and the improvement of public spaces. A multi-year evaluation by MJB found improvement in many of the key measures being evaluated in these neighborhoods: General satisfaction with the neighborhood Satisfaction with personal safety in the neighborhood Residents reporting a high level of social cohesion Parents who see their neighborhood as a good place to raise children 47% 55% 16% 61% 24% 56% 60% 68% 58% 68% 2012 2015 National Average 23% National Average 84% National Average 72% National Average 15% National Average 17% 7% Youths volunteering Adults volunteering 21 ELKA: JDC Institute for Leadership & Governance Merhav Meshutaf Merhav Meshutaf (Common Ground) was created within the framework of the Government-Civil Society Initiative, which was established in 2012 as a joint venture of seven government ministries, Diaspora Jewry, and the JDC-Israel Institute of Leadership and Governance. The initiative’s overarching goal is to enhance the interface between the government and civil society in order to strengthen social resilience in Israel. To further this goal, Merhav Meshutaf is cultivating change agents, a group of key personnel within government ministries and civil society organizations. Twenty-seven directors from government ministries and civil society organizations took part in the first round of the program, which ran from December 2015 to April 2016. An evaluation implemented by MJB found that: While participants from each sector harbored very negative and suspicious views toward participants from the opposite sector at the start of the program, they finished the program with a significant positive change in attitude. Many of the participants reported that they had already begun to implement what they learned in their work, particularly with regard to incorporating dialogue with the other sector during the planning and design phase of new projects. On the other hand, the study also revealed that the government participants felt less equipped with tools and less prepared to lead inter-sector projects than their civil society counterparts. They were also somewhat less inclined to see themselves leading inter-sector dialogue in their workplace. Participants also pointed to the need to include local government representatives in future cycles of the program, and the next cohort will indeed include significant representation from local governments (30%). A follow-up study of the graduates of all programs that aim to create agents of change as part of the Government-Civil Society Initiative will be implemented next year. -3 -2 -1 0 1 2 3 Suspicion Trust -3 -2 -1 0 1 2 3 Distance Closeness -3 -2 -1 0 1 2 3 Disrespect Appreciation GOVERNMENT: beginning end CIVIL SOCIETY: beginning end HOW PARTICIPANTS VIEWS CHANGE AS PROGRAM PROGRESSES 22 ESHEL: Elderly Needs Assessment for the Strategic Planning Process As a basis for Eshel ‘s five-year strategic planning process (2016-2020), the Myers-JDC-Brookdale Institute gathered information on key indicators relating to Israel’s elderly population (aged 65+), as well as the 55–64 age group that will become part of Eshel’s target population in the coming years. Among the key findings: Prevalence of serious health issues grows significantly with age of elderly men 25% 10% of elderly women Yet life expectancy is higher for women and their retirement age is currently lower. This has significant implications for Eshel since it now places more emphasis on elderly employment and better planning for retirement. Elderly men more likely to take part in the workforce Growing elderly population 2015 2035 (*projected) 80 70 60 50 40 30 55–64 65–74 75+ 35% 48% 62% As Israel’s population ages, overall needs will grow, and the service system will need to address this challenge. Israel’s elderly population will go from 939,000 in 2015 to a projected 1.66M in 2035, with the proportion of elderly in the total population going from 10.6% to 14.6%. 939,000 1.66 million* SERIOUS HEALTH ISSUES AGE 23 INTEGRATED CARE In Israel, as in other developed countries, the challenges of coordination, continuity of care, and an holistic approach to the needs of the elderly have become more salient as public services have evolved. An inter-ministerial steering committee formed to deal with this issue implemented a pilot program of integrative care for the elderly in one municipality in collaboration with Eshel. The goal of this initial pilot was to examine how to introduce a model of care management suitable for the Israeli service system. The pilot was accompanied by a study conducted by MJB to evaluate its implementation and help further develop the model. Among the main findings: The study reaffirmed that there is an elderly population that is not utilizing services in an optimal way due to service fragmentation and the absence of an holistic view of their needs. Based on the findings, it was decided to implement an adjusted program model in three additional municipalities, accompanied by an evaluation that will focus on outcomes for the elderly, their families, and the service system. Most participants have various social and economic difficulties, and 57% have no family support. About two-thirds of the participants received services that they had not received prior to the intervention, even though they were known to the system. 68% 60% of participants succeeded in implementing at least one recommendation given by a care coordinator in areas such as health, social services, daily activities, etc. The study raised important issues relating to the implementation of the model, such as: Need to create a clear address that participants understand they can contact in time of need. Need for a structured, systematic solution for monitoring participants and regularly assessing their needs. Need to clearly establish care managers’ responsibilities and provide them with further training. Most program participants suffered from multiple health problems: 60% have three or more serious health problems, and a third have seven or more. 24 Supported Housing in the Community for People with Disabilities In recent years, there has been growing recognition of the importance of increasing the opportunities for people with disabilities to make choices about their lifestyle. One major decision relates to the choice of living in independent housing rather than a parent’s home or an institution. But in order to move into independent housing, people with disabilities need significant preparation, assistance, and support. In 2012, a pilot was launched with three community organizations for Supportive Housing programs in three different areas of Israel. The program provides the participants with personal assistance in dealing with many aspects of their lives: finding an apartment and roommates, household management, employment, social and recreational activity, marital relationships, and contact with services in the community. An evaluation conducted by MJB Institute in 2014 examined the implementation and outcomes of the program in order to provide feedback for its improvement and as a basis for decisions regarding its expansion and dissemination. It included interviews with the participants and their parents, representatives of MOSAS (Ministry of Social Affairs and Social Services) and the local social service departments, program directors at the implementing organizations, and the care coordinators. Among the main findings: Participants who moved into independent housing during the program noted that it had helped them make the transition or had made the move easier and more successful. Participants who had not yet moved said that the program had helped them to believe that they were capable of moving and helped them to plan the move. One of the key issues debated by the program developers was whether to limit its duration in advance. According to the participants, setting a limit in advance could undermine their confidence in their ability to make the transition. Some felt that they would always need assistance, while others believed that over time they would need less and less support. In fact, several participants discontinued their participation once they had completed the transition and did not feel the need for further assistance. Based on the initial findings with respect to the assistance provided, a decision was made to proceed immediately to expand the program in a number of directions: 2. Target population: expand the target population to include people with sensory and intellectual disabilities. 3. Population groups: adapt the model for the ultra-Orthodox and Arab populations. 4. Areas of assistance: expand the areas of assistance in light of the diverse needs of the participants. 1. Geographic areas: expand the program into 12 areas 37 municipalities in Israel Unlimited 25 Tevet: Workforce Integration Employment Advancement Program Feedback from participants: Participants reported that the job advancement professionals’ support and guidance and the financial assistance toward studies that they received were the most important services offered by the program. Implementation: Job advancement professionals assumed responsibility for coordinating the program in addition to their regular work at the employment center. They reported that the increased workload and lack of funding limited their ability to allocate appropriate time to the participants. Based on the findings, improvements have been made to the program model, including assigning more full-time dedicated job advancement professionals. It was also decided to expand the program to provide effective advancement support services to a larger number of low-wage workers. The expansion will be accompanied by an RCT study. Signed up for study courses, mostly in vocational training. Increased their wages by 10% or more after two years. Of participants in the Eshet Hayil program for women increased their wages by 10% or more. 30% 52% 39% 56% 32% 34% JDC-Tevet develops programs to integrate populations with low labor force participation into the workforce. But while many program participants have succeeded in gaining jobs, they are often employed at minimum wage levels. Job advancement has therefore been adopted as one of Tevet’s strategic directions in its multi-year planning process. The Job Advancement Program was created in 2012 to help program participants increase their wages and improve their work conditions. Participants, aged 20-45, were given personalized work plans and guidance and support to implement those plans. Selected participants were encouraged to enroll in vocational training or academic studies, and they received financial assistance for their studies. MJB evaluated the implementation of the program in its first two years, tracking the employment outcomes of the participants against a comparison group of similar individuals who qualified for the program but did not participate. This served to assess the extent to which the job advancement found among participants could be attributed to the program. Among the main findings: Participants Comparison Group