| Notes |
15
JDC-Israel
GENERAL M&E UPDATE
Over the past year, and with generous support from the Harry and Jeanette Weinberg Foundation, JDCIsrael launched a major effort to upgrade its Monitoring & Evaluation (M&E) capacity, both within each of
its five divisions and across a number of strategic areas. This effort has been guided by a steering committee
led by Professors Gordon Berlin (MDRC) and David Weisburd (George Mason and The Hebrew University) and
Dr. Aviva Zeltzer-Zubida. This section describes the work done across JDC-Israel as well as within each of its
five divisions:
Ashalim: Children and Families at Risk
ELKA: JDC Institute for Leadership
and Governance
ESHEL: Elderly
Israel Unlimited: People with Disabilities
TEVET: Workforce Integration
16
The JDC-Israel Measurement & Evaluation Task Force
JDC-Israel established an M&E Task Force at the beginning of 2016, building on the M&E point persons
designated by each division three years ago as part of JDC’s global efforts to advance M&E throughout the
organization. The Task Force includes the divisional point persons, as well as a representative from the MyersJDC-Brookdale Institute (MJB), which plays a critical role in the development of tools and guidelines. It meets
for several hours each month.
The developments described below are the result of the Task Force’s focused efforts.
DESIGN
Identify a social challenge
Learn the issue
Map best practices in
Israel and abroad
Design and plan
the intervention
NURTURE
Launch a pilot
Test and evaluate
Refine model
Build capacity of
professionals and
infrastructure
ACCELERATE
Assist government
agencies and other
partners to scale up
and replicate nationally
Provide oversight,
ongoing consultation,
professional training, and
program development
Standardized DNA Criteria for All JDC-Israel Programs
JDC-Israel created the DNA (Design, Nurture, and Accelerate) model a few years ago to describe its
program development strategy.
The M&E Task Force was tasked with taking the model to the next level so it could be used as a tool to
conceptualize, plan, manage, and monitor the different stages of program development across JDCIsrael. But first, the different stages needed to be defined more precisely and standardized
criteria developed to gauge advancement from one stage to the next.
Together with MJB, the task force created a tool
that determines:
1. A program’s current stage of development
2. Clear criteria for when, if, and how it should advance to the next stage.
Task Force members have now begun integrating this tool into the work processes of their divisions.
In Eshel, Ashalim, and Tevet, detailed definitions of the steps required to implement each stage of the
DNA model were developed and the new workflows are now being disseminated across the divisions.
The divisions have begun training their staff on the DNA model, and a cross-JDC Israel seminar was
held in 2016.
17
Evaluation Policy
The M&E Task Force drafted a JDC-Israel Evaluation
Policy putting the global M&E principles introduced
by JDC in 2012 into the local context. The policy has
been approved by the JDC-Israel Management Team
and includes:
Goals and aims of M&E in JDC-Israel programs
Basic standards for all evaluation studies of
JDC-Israel programs
Specific M&E objectives that need to be
accomplished at each stage of the DNA process
Organizational capacities needed to support M&E
in JDC-Israel
Strategic partnership on M&E with Myers-JDCBrookdale Institute, and clarification of its role
vis-a-vis external evaluation
The evaluation policy lays out a number of
strategic directions that will shape M&E work
in the upcoming years:
1. Greater emphasis on evaluation of outcomes–
both at the level of individual programs, but more
importantly at the level of an area, a division, and
across JDC-Israel. This includes defining shared
outcome goals as well as shared measures.
2. Developing data systems for ongoing outcome
measurement
3. Promoting more rigorous evaluation studies
4. Investing in long-term outcome studies in order to
measure impact after programs have been handed
over to the government
A number of steps in these directions have been taken
by the different divisions:
SHARED OUTCOMES AND MEASURES
The M&E task force is in the process of developing key
JDC Israel-wide shared goals and outcome measures
that will serve as the basis for evaluating all JDC-Israel
programs. As JDC-Israel’s mission is to improve the
system of care for Israel’s most vulnerable populations,
these outcome measures relate both to the vulnerable
populations and to the necessary changes in the
system of care.
These JDC-Israel outcomes serve as the foundation
for developing a set of shared measures for programs
for children and families (Ashalim), for elderly
(Eshel), for people with disabilities (Israel Unlimited),
and for programs that promote integration into the
workforce (Tevet).
In Ashalim, a “program success team” was created
to identify key crosscutting program outcomes and
measures for children and youth, families, and the
community. The team prioritized measurements that
are already in use by Ashalim partner ministries
and/or were validated and are well known in the
international literature.
Both Eshel and Israel Unlimited have launched
a similar process to define crosscutting program
outcomes and develop shared measures in
collaboration with MJB.
Tevet, working with MJB, has taken a major step
forward by launching a process to develop shared
measures for employment together with all relevant
government agencies. These shared measures
would serve as the standard for evaluation of all
government-funded employment programs in Israel.
In 2016, Tevet, MJB, and the Ministries of the Economy
and Social Affairs reached an agreement on measures
that relate to key employment outcomes such as
integration into the work force, job placement, and
job retention.
18
2017 will serve as a pilot period for experimenting with
the integration of these measures into the work of the
different partners. The inter-organizational working
group will follow the implementation of these measures
with the aim of deciding on their permanent use. The
group will also begin developing shared measures next
year for additional employment outcomes, such as job
advancement, soft-skills, and employability.
Within Tevet, the shared measures are being
integrated into the ongoing measurement system
and system reports are being adjusted accordingly.
The shared measures will also be used in all
forthcoming evaluations.
ONGOING OUTCOME
MEASUREMENT
Tevet utilizes an ongoing outcome measurement
system, called Spot, in all its pilot programs. Many
programs continue to use that system even after they
have been transferred to the government for full-scale
implementation. In the past year, Tevet began reviewing
the requirements of the Spot system with the aim of
developing a more updated model.
Ashalim will pilot the data system developed by the
National Program for Children and Youth at Risk in
five Ashalim programs. If this proves successful, and
based on the lessons learned, it plans to integrate this
data system into most Ashalim programs in the coming
years.
As part of the global JDC initiative, similar systems are
currently in development in all other divisions. These
systems will integrate all relevant program materials,
serve as the exclusive database for programs, and
support data-informed decision-making processes and
ongoing program monitoring.
PROMOTING RIGOROUS
EVALUATION IN JDC-ISRAEL
To improve the sophistication and rigor of evaluation
studies conducted on behalf of JDC-Israel, each division
is seeking to expand the use of control groups and
experimental research designs, including randomized
control trials (RCT), where appropriate. A number of
RCTs are now in active planning in Tevet, Eshel, and
Ashalim and will be implemented starting next year.
While some JDC programs have too many changing
variables or other factors hindering the ability to
implement experimental research designs, it is our
intention to continue to promote rigorous evaluation
studies in order to best support learning and decisionmaking processes.
CROSSCUTTING AND LONG-TERM
IMPACT STUDIES
The Institute for Leadership and Governance has
decided to initiate follow-up studies that examine
longer-term outcomes of capacity building programs
as well as holistic studies that examine the impact of
a portfolio of interventions that are meant to move
the needle on a shared challenge.
Tevet has launched two major efforts in collaboration
with MJB: an evaluation of the integration and
expansion of Tevet programs by the government after
the completion of their pilot stage, and an assessment
of long-term employment outcomes for Tevet
graduates based on data from the Central Bureau
of Statistics.
19
106
external evaluations
Close to
30%
used some sort of
comparison group
Breakdown
by division
Breakdown
by type
27%
Eshel
23%
Tevet 20%
Ashalim
18%
IU
18%
ELKA
20%
process
70%
process &
outcome
10%
outcome
The following pages highlight program findings from flagship
programs in each of the five JDC-Israel divisions.
PROGRAM FINDINGS
All of JDC-Israel’s programs are evaluated at some point in their life cycle. In the past 6 years there have been:
20
Ashalim: Children and Families at Risk
Better Together
Better Together is a neighborhood change program
that promotes the creation of effective communitybased systems in which professionals and residents
work together to improve the wellbeing of children
and youth in Israel’s disadvantaged communities and
reduce their exposure to risk situations.
Since 2006, the program has been implemented
in some 43 communities with a broad range of
ethnic backgrounds. Ashalim serves as the backbone
organization in these communities and promotes
an organized process of networking and learning.
In the belief that it does indeed take a village to
raise a child, Better Together emphasizes three
levers of change:
Voice and involvement of the residents—
from a passive to a proactive community
Integrated multi-disciplinary and multiorganizational mechanisms—from silos to
coordination and cooperation among key
communal organizations and their leaders.
This facilitates a comprehensive approach to
addressing challenges.
Responses for children and youth—from
fragmented services to a service continuum
Program interventions and activities are chosen
to fit the specific needs of each neighborhood.
They include one-time special events to increase
awareness of the program and mobilize resident
support and engagement, and ongoing initiatives
such as after school learning centers, leisure
activities, and the improvement of public spaces.
A multi-year evaluation by MJB found improvement
in many of the key measures being evaluated in
these neighborhoods:
General
satisfaction
with the
neighborhood
Satisfaction
with personal
safety in the
neighborhood
Residents
reporting a
high level
of social
cohesion
Parents who
see their
neighborhood
as a good place
to raise children
47% 55%
16%
61%
24%
56%
60% 68% 58% 68%
2012 2015
National
Average
23%
National Average
84%
National Average
72%
National Average
15%
National Average
17% 7%
Youths
volunteering
Adults
volunteering
21
ELKA: JDC Institute for Leadership & Governance
Merhav Meshutaf
Merhav Meshutaf (Common Ground) was created
within the framework of the Government-Civil Society
Initiative, which was established in 2012 as a joint
venture of seven government ministries, Diaspora
Jewry, and the JDC-Israel Institute of Leadership and
Governance. The initiative’s overarching goal is to
enhance the interface between the government and civil
society in order to strengthen social resilience in Israel.
To further this goal, Merhav Meshutaf is cultivating
change agents, a group of key personnel within
government ministries and civil society organizations.
Twenty-seven directors from government ministries
and civil society organizations took part in the first
round of the program, which ran from December
2015 to April 2016.
An evaluation implemented by MJB found that:
While participants from each sector harbored
very negative and suspicious views toward
participants from the opposite sector at the start
of the program, they finished the program with
a significant positive change in attitude.
Many of the participants reported that they had
already begun to implement what they learned in
their work, particularly with regard to incorporating
dialogue with the other sector during the planning
and design phase of new projects.
On the other hand, the study also revealed that the
government participants felt less equipped with
tools and less prepared to lead inter-sector projects
than their civil society counterparts. They were also
somewhat less inclined to see themselves leading
inter-sector dialogue in their workplace.
Participants also pointed to the need to include
local government representatives in future cycles
of the program, and the next cohort will indeed
include significant representation from local
governments (30%).
A follow-up study of the graduates of all programs
that aim to create agents of change as part of
the Government-Civil Society Initiative will be
implemented next year.
-3 -2 -1 0 1 2 3
Suspicion Trust
-3 -2 -1 0 1 2 3
Distance Closeness
-3 -2 -1 0 1 2 3
Disrespect Appreciation
GOVERNMENT: beginning
end
CIVIL SOCIETY: beginning
end
HOW PARTICIPANTS VIEWS CHANGE
AS PROGRAM PROGRESSES
22
ESHEL: Elderly
Needs Assessment for the Strategic Planning Process
As a basis for Eshel ‘s five-year strategic planning process (2016-2020), the Myers-JDC-Brookdale Institute
gathered information on key indicators relating to Israel’s elderly population (aged 65+), as well as the 55–64
age group that will become part of Eshel’s target population in the coming years.
Among the key findings:
Prevalence of serious health issues grows significantly with age
of elderly men
25% 10%
of elderly women
Yet life expectancy is higher for women and
their retirement age is currently lower. This has
significant implications for Eshel since it now
places more emphasis on elderly employment and
better planning for retirement.
Elderly men more likely to take
part in the workforce
Growing elderly population
2015
2035 (*projected)
80
70
60
50
40
30
55–64 65–74 75+
35%
48%
62%
As Israel’s population ages, overall needs will grow, and the service system will need to address this challenge.
Israel’s elderly population will go from 939,000 in 2015
to a projected 1.66M in 2035, with the proportion of
elderly in the total population going from 10.6% to 14.6%.
939,000
1.66 million*
SERIOUS HEALTH ISSUES
AGE
23
INTEGRATED CARE
In Israel, as in other developed countries, the
challenges of coordination, continuity of care,
and an holistic approach to the needs of the elderly
have become more salient as public services
have evolved.
An inter-ministerial steering committee formed to
deal with this issue implemented a pilot program of
integrative care for the elderly in one municipality in
collaboration with Eshel. The goal of this initial pilot
was to examine how to introduce a model of care
management suitable for the Israeli service system.
The pilot was accompanied by a study conducted by
MJB to evaluate its implementation and help further
develop the model.
Among the main findings:
The study reaffirmed that there is an elderly
population that is not utilizing services in an
optimal way due to service fragmentation and
the absence of an holistic view of their needs.
Based on the findings, it was decided to implement
an adjusted program model in three additional
municipalities, accompanied by an evaluation that
will focus on outcomes for the elderly, their families,
and the service system.
Most participants
have various social
and economic
difficulties, and
57%
have no family
support.
About two-thirds of the participants
received services that they had not
received prior to the intervention, even
though they were known to the system.
68%
60%
of participants succeeded in
implementing at least one
recommendation given by a care
coordinator in areas such as health,
social services, daily activities, etc.
The study raised important issues relating to the
implementation of the model, such as:
Need to create a clear address that participants understand
they can contact in time of need.
Need for a structured, systematic solution for monitoring
participants and regularly assessing their needs.
Need to clearly establish care managers’ responsibilities
and provide them with further training.
Most program
participants suffered
from multiple health
problems: 60% have
three or more serious
health problems, and
a third have seven
or more.
24
Supported Housing in the
Community for People with
Disabilities
In recent years, there has been growing recognition
of the importance of increasing the opportunities for
people with disabilities to make choices about their
lifestyle. One major decision relates to the choice of
living in independent housing rather than a parent’s
home or an institution. But in order to move into
independent housing, people with disabilities need
significant preparation, assistance, and support.
In 2012, a pilot was launched with three community
organizations for Supportive Housing programs
in three different areas of Israel. The program
provides the participants with personal assistance
in dealing with many aspects of their lives:
finding an apartment and roommates, household
management, employment, social and recreational
activity, marital relationships, and contact with
services in the community.
An evaluation conducted by MJB Institute in 2014
examined the implementation and outcomes of
the program in order to provide feedback for its
improvement and as a basis for decisions regarding its
expansion and dissemination. It included interviews
with the participants and their parents, representatives
of MOSAS (Ministry of Social Affairs and Social
Services) and the local social service departments,
program directors at the implementing organizations,
and the care coordinators.
Among the main findings:
Participants who moved into independent housing
during the program noted that it had helped them
make the transition or had made the move easier
and more successful. Participants who had not yet
moved said that the program had helped them to
believe that they were capable of moving and helped
them to plan the move.
One of the key issues debated by the program
developers was whether to limit its duration in
advance. According to the participants, setting a limit
in advance could undermine their confidence in their
ability to make the transition. Some felt that they
would always need assistance, while others believed
that over time they would need less and less support.
In fact, several participants discontinued their
participation once they had completed the transition
and did not feel the need for further assistance.
Based on the initial findings with respect
to the assistance provided, a decision was made
to proceed immediately to expand the program
in a number of directions:
2. Target population: expand the target
population to include people with
sensory and intellectual disabilities.
3. Population groups: adapt the model for
the ultra-Orthodox and Arab populations.
4. Areas of assistance: expand the areas
of assistance in light of the diverse needs
of the participants.
1. Geographic areas:
expand the program into
12
areas
37
municipalities
in
Israel Unlimited
25
Tevet: Workforce Integration
Employment Advancement Program
Feedback from participants: Participants reported
that the job advancement professionals’ support
and guidance and the financial assistance
toward studies that they received were the most
important services offered by the program.
Implementation: Job advancement professionals
assumed responsibility for coordinating the
program in addition to their regular work at the
employment center. They reported that the
increased workload and lack of funding limited
their ability to allocate appropriate time to the
participants.
Based on the findings, improvements have been
made to the program model, including assigning
more full-time dedicated job advancement
professionals. It was also decided to expand the
program to provide effective advancement support
services to a larger number of low-wage workers.
The expansion will be accompanied by an RCT study.
Signed up for study
courses, mostly in
vocational training.
Increased their wages by
10% or more after two years.
Of participants in the Eshet Hayil
program for women increased
their wages by 10% or more.
30%
52% 39% 56%
32% 34%
JDC-Tevet develops programs to integrate
populations with low labor force participation
into the workforce. But while many program
participants have succeeded in gaining jobs, they
are often employed at minimum wage levels.
Job advancement has therefore been adopted as
one of Tevet’s strategic directions in its multi-year
planning process.
The Job Advancement Program was created in 2012
to help program participants increase their wages
and improve their work conditions. Participants, aged
20-45, were given personalized work plans and
guidance and support to implement those plans.
Selected participants were encouraged to enroll in
vocational training or academic studies, and they
received financial assistance for their studies.
MJB evaluated the implementation of the program
in its first two years, tracking the employment
outcomes of the participants against a comparison
group of similar individuals who qualified for the
program but did not participate. This served to assess
the extent to which the job advancement found
among participants could be attributed to
the program.
Among the main findings:
Participants
Comparison Group |