Reinventing Education

IBM’s Reinventing Education Grant Partnership Initiative __________ Individual Site Reports CCT Reports March 2001 Robert Spielvogel Cornelia Brunner Shelley Pasnik Julie Thompson Keane Wendy Friedman Laura Jeffers Karen John Julia Hermos  Table of Contents Highlights of the Research on Implementation Background to Reinventing Education About This Study Methodology Broward Charlotte-Mecklenburg Chicago Philadelphia San Jose San Francisco Unified School District Vermont West Virginia Appendix 1: Watch-me!-Read Evaluation .......... 1 ......... 3 ......... 5 ......... 6 .......... 11 ......... 21 ......... 29 ......... 36 ......... 46 ......... 56 ......... 63 ......... 69 ......... 84  Highlights from the Implementation Study Beginning in 1994 a new initiative began that creates working partnerships between major urban school districts, rural states and the IBM Corporation. Reinventing Education is the name of this initiative, due in part to IBM's Chairman and CEO, Louis V. Gerstner, Jr.'s book on educational transformation of America's public schools. Given the ambitious goals of these partnerships in changing key aspects of teaching and learning, the initiative is aptly named. This summary report culminates three years of studying the implementation of the Reinventing Education program at the districts and states that are participating in its implementation. The report summarizes what has evolved at the sites and presents evidence of what is changing while examining common challenges and success factors across the sites. Key Outcomes during the implementation phase: • Unlike many education reform initiatives, the solutions that directly address student learning through the provision of new or improved forms of instruction have had significant positive impact on student achievement in grades 7 through 11 in mathematics, language arts, social studies and science and on the development of early reading skills. Teachers’ use of juried lesson plans in the core disciplines of mathematics, language arts, social studies and science developed with the Instructional Planner solution result in significant gains in performance for students in grades 7 through 11 performing at below average levels while higher performing students using these lessons maintain their performance edge. A study conducted by reading researchers at Vanderbilt University of the Watch Me Read program in use with beginning readers in first grade found significant and consistent improvements is reading level, word recognition, and comprehension in comparison to similar students not using this application. • The solutions go beyond technology – they address process and change on an organizational level within districts and states that is a fundamental component of a reform effort, given the scope and time frame over which the changes occur. The solutions are not single applications but rather an enterprise-level platform and team of knowledgeable people within the district that can use the platform to address specific issues or challenges facing the district or state. This combination of flexibility and experience allows those sites that participated in the Reinventing Education initiative to continue leveraging their investment by increasing both the scale of their implementation in addressing the original issue and the broadening the scope of educational issues to which they are applying the solutions. As a result, these sites are using technology to address specific issues related to educational reform and improvement. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 1  • The Reinventing Education Initiative evolved into a long-term partnership between IBM and these districts and states. The involvement between IBM and the sites started with a detailed planning period, moved through an extended period of iterative development, feedback, refinement, and testing, and matured into a significant scale up effort at the sites where the solutions became institutionalized and part of the district or state’s ongoing efforts. This phase took three to five years. Even after the grant period ended, IBM remains involved with the sites and continues the partnership relationship. This level of commitment is rare for school-based technology-oriented investments and is tangible evidence of the partnerships’ commitment to fundamental integration of technology- assisted solutions rather than just implementation of new technology applications. • Many of the sites have made significant changes in the way ongoing professional development of teachers is conducted. All of the sites have evolved sophisticated ways of providing professional development for the teaching staffs in the schools working with these solutions. For some of the sites, this was the main focus of their solution; others developed professional development as part of the implementation process. All evolved professional development solutions characteristic of what recent research tells us are key qualities for effective teacher development – sustained opportunities that are embedded in the regular teaching experience and that are able to be used immediately by the participants. • The sites funded in the Reinventing Education I and II initiatives that moved to implementation are now running the solutions on their own, substantially increasing their investment and commitment to its growth, and expanding to deploy other solutions created on the platform at other sites: change in leadership. These are all indicators of the RE solutions successfully moving from innovative experiments to core, systemic components of institutional operations. Within the context of educational reform in general, and technology-based innovation specifically, this level of commitment in terms of time and scale is unusual. Externally-funded grants in schools encounter significant barriers that often keep their solutions on the periphery and in a fragile state that does not continue in any significant way after the grant funding is completed. This level of success is unusual. As noted in the recently published report “Learning From Each Other” (Konley, 2000) from the Grantmakers for Education, relatively few fuders have engaged school districts in long-term change endeavors and the crusade for better schools has been long and difficult, often frustrating and disappointing. The report shares lessons learned from some of the larger-scale reform efforts in school districts. The Reinventing Education Initiative adds another dimension to our understanding: what does it take to realize technology’s potential for improving teaching and learning. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 2 Background to Reinventing Education The Reinventing Education partnerships have developed a wide range of applications using new information and communications technologies. All of them support new ways for teachers, administrators, parents, and students to work toward overcoming specific barriers to higher performance. In the form of grants and ongoing support from the company’s corporate community relations program, IBM has invested over $45 million in Reinventing Education (RE). More notable than the size of this corporate philanthropy is its substance: rather than just supply cash or hardware and software donations, the IBM Corporation has used the investment to engage IBM researchers, technical talent, and consulting services, along with smaller contributions of hardware, in addressing significant educational problems facing public schools. And for their part, the participating school districts and departments of education have served as ongoing partners, using the support they have received to make the co-creation of their solution an embedded part of their organization. Although the work occurring at each Reinventing Education site is as diverse as the locations are dispersed — the initiative spans the country from California to Florida and from Vermont to Texas — the overarching goal of the program has been to create lasting change leading to higher student achievement. Whether it is a model of continuous professional development that uses technology to facilitate teacher-to-teacher support, a data warehouse application that delivers timely student information to educators’ desktops or an online system to increase parents’ participation in their children’s education, each solution is designed to break down a systemic barrier to high academic achievement identified by each site. All of these sites were selected through a competitive Request for Proposals process that looked for a commitment to change and for identified barriers that were likely to be relevant to other districts and states involved in systemic change. To date, the Reinventing Education initiative has included two distinct phases of support.1 Beginning in 1994, in response to grant proposals submitted by school districts and state departments of education, IBM gave approximately $2 million awards each to10 individual sites to create customized solutions. Nine of these sites, which we have listed in Chart 1 at the end of this report, went into full implementation. In October 1997, IBM awarded 12 additional grants as part of Reinventing Education 2. These grants were focused on the adoption and adaptation of some of the emerging solutions from the RE1 sites and thus were smaller financially and shorter in duration: each award ranged from $475,000 to $875,000 and the grant period for the second round of grants officially lasted approximately eight months. Despite the tighter timeline and smaller funds, RE2, like RE1, was intended to forge an educational partnership between 1 Brazil, Ireland, Italy, Vietnam, Singapore, and the United Kingdom. IBM also has extended the Reinventing Education initiative abroad, awarding grants in Rio de Janeiro, IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 3 IBM and the grant recipient. As of December 2000, nine of the 12 sites, which are listed in Chart 2 at the end of this document, had entered full implementation; the remaining three are making steady progress toward implementation. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 4 About This Study The Center for Children and Technology (CCT) is based in New York City and for over twenty years has specialized in conducting research and evaluations on the roles that technology can play in improving teaching and learning.2 CCT is a division of the Education Development Center, Inc, which is one of the largest not-for-profit agencies devoted to bridging research in practice in areas of education, health, and human development within the United States and throughout the world.3 CCT was contracted by IBM to conduct a three-year evaluation of the Reinventing Education initiative. This evaluation, which began in spring 1998 and ran through December 2000, builds upon earlier research we performed in Winter 1997, which examined the planning process and early pilot work undertaken by each Reinventing Education site.4 This Report picks up from that point and looks at the Reinventing 1 sites during the period in which they moved from initial pilots to full scale implementation and it offers a preliminary look at the Reinventing 2 sites that adopted some of the emerging solutions. The description of the process through which each project went in building upon its partnership with IBM, and the evidence of each project’s institutionalization, is central to this report. Although we paid close attention to the many particulars that characterized the education reform efforts in each site —from the political climate to the geographic location, and from the way each school and district was structured to the work styles that project leaders brought to their jobs — we also looked for elements that we could generalize. As such, this report is intended not only for those whose work we have evaluated but for others engaged in education reform initiatives, especially those involving technology integration, more broadly. We have gathered important evidence about how technologies can be enlisted to address a range of difficult problems in a variety of settings, and have attempted to arrange the sections from the general to the specific. 2 3 4 Information about CCT can be found at http://www.edc.org/CCT More information can be found at http://www.edc.org See the 1997 report at www.ibm.com/ibm/ibmgives/grant/education/programs/reinventing/cct.html IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 5  Methodology Documenting the impact of technology on some specific aspect of learning is difficult except in the most isolated of cases where all of the confounding variables can be controlled. The Reinventing program works in the opposite environments - complex urban and state systems where change programs must work in concert to have any impact at all. In these situations, as in any school-level or district-level program, technology is rarely an independent variable; instead it works either as a catalyst or an amplifier leveraging or enhancing other efforts. Despite the large public and private investments in educational technology, there is considerable concern that we are not producing enough evidence to justify and to focus this investment. In the spring of 2000, CCT participated in an invitational symposium sponsored by the US Department of Education to examine best research practices from a variety of disciplines that can be harnessed in documenting technology's contribution to improved teaching and learning.5 As external evaluators for a program as ambitious as Reinventing, we face two challenges in examining impact at each site: was the solution actually implemented at some level sufficient enough for change to occur and what are the likely indicators that should show signs of change in the near term if the solution is working? The first challenge was summed up nicely at that symposium by Dr. Eva Baker of the Center for Research on Evaluation, Standards and Student Testing when she made a telling comment about evaluating instructional and educational technology projects. She said that anyone working in the field of technology education could write, site-unseen, the evaluation summary of almost any current project. That evaluation would say, “This project has real potential if only it had been implemented more fully.” In thinking about Reinventing Education's impact, identifying and tracking indicators of institutionalization of the solution at each site was one of the key components of our work over the last three years. Given the complexity and size of these schooling environments, measures of implementation are a prerequisite for looking at effects. The second challenge came in unpacking the relationship between each of these solutions and the ultimate goal of the entire program: higher performance as indicated by higher student achievement. Our most public current measure of student achievement, the standardized test, is a tertiary measure. It is a broad indicator of a combination of many factors and not sensitive to registering more immediate influences. While it remains a key measure of the ultimate success of the program, only a few of the solutions, notably the West Virginia lesson plans and Watch Me! Read, directly change what occurs in student learning in a manner likely to have short-term impact on standardized test scores. The other solutions, if successful, will ultimately contribute to system-wide changes in this variable but indirectly and only after an extended period of time that exceeds the time frame of this evaluation. We needed to identify interim indicators that should show change within shorter term and that are known to be contributors to higher performance. 5 The synthesis of that work can be found at http://www.sri.com/policy/designkt/found.html. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 6  The methodology guiding our evaluation, therefore, included three distinct but related phases: Phase 1: Phase 2: We visited each RE site, interviewed key leaders from IBM and the participating schools and districts, examined all internal documentation pertaining to the partnership and met with representatives from participants in the project who were likely users of the technology tools being developed. We assembled a chronology of important events that occurred prior to out visit and recorded project leaders’ expectations of and desired outcomes from the partnership. Based on what we learned in Phase 1, we drafted a set of indicators against which to measure the progress of the initiative. We focused on primary and secondary indicators that were likely to reflect the kinds of change that we could document within the timeframe of this study. In a few cases, this included standardized test scores. In others it was documenting changes in teaching practices or school-parent contacts that are known to be beneficial to improving student performance. Although there were variations from one set of indicators to the next, institutionalization of the project was a key indicator for each site. If the solution is treated as an isolated project or even as a demonstration model or proof-of-concept by the district or state, it will fail to achieve its potential impact on a systemic level. Many educational innovations fail to take hold after the initial phase of funded development. Others do not have the institutional commitment to survive changes in key stakeholders or remain limited in their adoption. Therefore the indicators that track the transfer of responsibility and ownership of the solution within the district and state and reveal evidence of its incorporation into mainstream infrastructure were critical as they often were prerequisites to larger scale indicators of specific impact on teaching practices and learning outcomes. After the project leadership agreed upon the set of indicators, we devised a workplan and instrumentation. We used these to collect data that could document the impact these solutions are having on teaching and learning in the educational communities where they are being implemented. During Phase 3, we collected evidence of success in terms of achieving the solution's intended benefits (and, in some cases, unintended benefits), and the current challenges that the solution faces. We did this through entrance and exit interviews, personal communication with project leadership, written and electronic survey instruments, utilization data, and, where relevant, existing institutional measures including test scores. Members of the research team conducted numerous visits to each of the sites over the three-year period, attending teacher training sessions, meeting with key IT personnel and project leaders, observing classrooms, and conducting interviews with state and district IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 7 Phase 3: leaders and community partners. Because of the duration of the evaluation, in many instances, evaluation team members formed close working relationships with project leadership on both the IBM and school-side. In addition to employing formal data collection instruments, the evaluation teams engaged in on-going communication with project leaders, sending e-mail messages back and forth and discussing new developments by phone as they arose. The focus of our evaluation was not exclusively on technology. We were not trying to imply a causal relationship between technology integration and student learning. Instead, we have examined the myriad ways schools districts and state departments of education are using technology as a lever to start a number of processes going, and how they were using technology to support reform efforts already underway. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 8 Chart 1: Reinventing Education 1 Site Goal Tool • Broward County Public Schools Promote accountability Data Warehouse • Charlotte-Mecklenburg Public Improve parent Wired for Learning Schools involvement and teacher professional development • Chicago Public Schools Improve teacher Wired for Learning professional development • Cincinnati Public Schools6 Promote accountability Credit Granting Standards Strengthen assessment Tracking Tool • School District of Philadelphia Improve communication Continuous Practice and teacher professional Improvement Model development • San Francisco Unified School District Increase student success in the general education program Improve the Special Education referral process Increase teacher professional development and parent involvement Student Support Team Tool • San Jose Unified School Improve teacher Electronic Portfolio District professional development Tool/Wired for Learning and classroom practice • Vermont Department of Strengthen curriculum and Authentic Assessment Education assessment Tool/Wired for Learning • West Virginia Department of Improve teacher Instructional Planner/ Education professional development Wired for Learning and classroom practice 6 the implementation phase as of this report. Therefore their progress is not covered in detail in herein. Cincinnati Public Schools started work on their program much later than the others and is just beginning IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 9 Chart 2: Reinventing Education 2 Site Goal Tool • Atlanta Public Schools Strengthen curriculum Visual Venture • Boston Public Schools Strengthen curriculum and Authentic Assessment assessment practices Tool • Detroit Public Schools Increase parent Wired for Learning involvement and teacher professional development • Durham Public Schools Increase parent Wired for Learning involvement and teacher professional development • Maryland State Department of Strengthen curriculum and Public Access Viewer Education assessment practices • Houston Independent School Strengthen curriculum Watch-me!-Read District • Memphis City Schools Strengthen curriculum and Authentic Assessment assessment practices Tool/Wired for Learning • New York City Public Schools Strengthen curriculum and Authentic Assessment assessment practices Tool • New York State Education Improve teacher Wired for Learning Department professional development and classroom practice • Rochester Public Schools Strengthen curriculum Visual Venture • South Carolina Department of Promote accountability Data Warehouse Education • Texas Education Agency Promote accountability Statewide Data Feed IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 10 Broward County Public Schools District Background Broward County Public Schools, Florida, with headquarters in Fort Lauderdale, is the fifth largest school district in the United States. It is also one of the fastest growing, with approximately 6,000-10,000 new K-12 students enrolled every school year. 2000-01 Data ü Number of K-12 Students: 249,923 û A diverse multicultural population from 152 countries, speaking 52 languages. û More than 53,000 students with special needs, including 6,632 gifted students. ü Number of Schools: û Elementary û Middle û High û Other ü Number of Instructional Staff: Budget: Solution Summary 215 128 35 24 28 13,110 $2.2 Billion The Broward County Reinventing Education project aimed to develop a technological solution to the challenge of both storing and making accessible longitudinal data on individuals within the district's school system to facilitate decision making and strategic planning. The necessity of such a tool, commonly referred to as the data warehouse, was based on numerous internal and external factors, from an unusually high student mobility rate to the need to respond to accountability pressures within the state of Florida. The ultimate impetus for the project was the need to improve student performance, and the district believed that efficient access to information on students would facilitate decision making within schools, leading to data-driven curriculum planning and professional development. The Center for Children and Technology’s (CCT) evaluation focused on monitoring the impact of the data warehouse on decision-making processes and on information dissemination within schools. We also gathered data around access to the technology and training needed for teachers and administrators to directly query the data warehouse, and how prepared they felt once they have received that training. Focus of Evaluation The evaluation team from CCT identified the following features of the program for use as indicators of success over the course of its evolution. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 11 • Differences between the ways teachers use data received from the district and the ways they use data retrieved through the data warehouse. • Differences between the ways school administrators use data received from the district and the ways they use data retrieved through the data warehouse. • Differences between the ways district level administrators use data received from the district and the ways they use data retrieved through the data warehouse. • An increase in access to and comfort around using technology at the school level. • How resources are allocated to Educational Technology Services (ETS) in order to maintain and further develop the solution. • An increase in the dissemination of information about and awareness of the data warehouse solution Current Status The Broward County Reinventing Education program has successfully developed a data warehousing system that stores a diverse array of information on students enrolled in the county's public school system. This information, furthermore, is available to an increasingly larger set of stakeholders in a growing number of formats for use in various contexts. The program initially targeted classroom teachers as its primary user audience, and trained them to use the Brio interface that gave access to data stored within the warehouse. This interface was extremely difficult for new technology users to work with, and required a considerable amount of user practice before the system could be mastered — a challenge for teachers who only need to access it once or twice a year. After refocusing its attentions in the direction of school administrators, ETS began to work closely with many of these administrators in order to understand what the schools' needs were regarding data and design. The product of these efforts is that the data warehouse system has accommodated new kinds of data, has created multiple mechanisms for making that data available in different formats, and is continuing to work with school- based users to further address their needs. With the availability of data to schools has come an understanding on the part of the district that administrators and teachers need support not only in accessing, but in interpreting information in order to make informed decisions regarding their students. These needs are currently being addressed through data analysis trainings and through the development of programs that will help schools learn to use mined data to create school improvement plans. Evidence of Benefits Professional Development 7 • The district is planning on future uses of the data warehouse in relation to professional development efforts and school improvement plans. The district's 7 All references to comments and findings are based on interviews conducted with teachers and administrators conducted in spring 1999 and spring 2000 and with ETS and district staff throughout the course of the evaluation. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 12  curriculum department is using data obtained through the warehouse to determine which integrated learning systems should be recommended for a particular school based on its students' performances. Furthermore, in order to help schools more effectively plan their own improvement efforts, programs are being developed that will train school personnel to interpret reports and develop strategic plans based on the findings. Lastly, as ETS returns to teachers as a target user group, the curriculum department is considering plans to organize them into learning communities that can focus on data analysis in order to use the information to develop their own classroom strategies. Effective Administration • School administrators now have access to a wealth of information in a timely and continuous manner from any location within the district, enabling them to plan for and address student needs more directly and efficiently. Principals and Assistant Principals repeatedly made comparisons between waiting months for long, indecipherable reports on student testing and the ease with which they are now able to quickly access the information when it arrives from the testing service and is entered into the warehouse. Furthermore, not only are these administrators having more immediate access to information, they are also able to look at the scores of particular groups of students based on demographic and behavioral information to further understand the patterns they are seeing, e.g., such as by absentee rate. Administrators also report that they are using test scores to set achievement goals for the year, and to think about professional development for their teachers, though these more complex applications are not yet ubiquitous. Access to data, and the ability to manipulate and compare this data, is also giving school personnel the ability to test past theories or "hunches." For example, one administrator explained that for years she had believed in tracking because she felt that it led to higher student achievement. Now, she said, the results are showing that mixed classes make for stronger student performance and that access to data is "proving or disproving different philosophies and theories." • The data warehouse has successfully brought information previously housed in disparate databases and accessible only through limited channels into one centrally located, widely accessible system, thus putting the power to access this information directly in the hands of the schools. Prior to the creation of the data warehouse, data used by schools was sent to them each year in the form of a long, printed report that showed students' test scores. Most often, according to school administrators, these reports arrived long after the time that the tests were taken, and after important planning decisions were made. In addition, these reports represented only a limited amount of information, and access to further data required that administrators submit separate requests and wait long periods of time for the information to be collated. The data warehouse has put the power to access this information directly in the hands of the schools, and requires only that personnel there understand how to use Brio and have access to the warehouse through a computer. For more complicated queries, schools are able to directly contact ETS, who will run queries and email schools the subsequent reports. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 13 • The data warehouse system, and the Brio interface through which its contents are accessed, has enabled users to collate data according to numerous parameters and organize data into various formats rather than receiving a single report of information that could not be further manipulated and reorganized. Reports supplied by the district in the past contained only one year of test data, making longer-term analyses impossible. Furthermore, the format of the reports, i.e., hardcopy, meant that no manipulations or variable comparisons, such as organizing by demographics, could be made without requesting a separate report, or hand entering the information into a new database. Because the data warehouse contains not only test scores, but also student attendance records, demographic information, and behavioral data, analyses can be run at schools that enable users to look for trends by various categories. The results of these analyses are nearly immediate, doing away with the long turnover times that schools previously had to endure, and can be presented in numerous formats, such as pie charts and graphs. Improved Communications • The district has consistently worked to make the data warehouse more accessible to more people by revamping the user interface, and is currently working on making the tool, along with pre-designed queries, available through the Web. ETS is extremely cognizant of the difficulty faced by users, particularly those who are not high users of technology generally, learning to access information with Brio. In response to this, they have worked to have Brio updated and redesigned over time, and have taken further steps to make it available through both PC and Mac platforms. Currently, ETS has launched a Web-based interface to the data warehouse, which includes pre-designed queries that focus on those reports teachers most often use in planning to address significant learning targets as determined by the needs of their class roster. As their roster changes during the school year with students moving in and out of their classes, teachers can instantly update the analysis to ensure that teaching is focused on student needs. The web interface makes this data even more accessible to teachers and allows the district to focus training on the interpretation and utilization of the information instead of the mechanics of working with a specialized software client. In addition to general accessibility and interface issues, ETS has recognized that different users are accustomed to having data presented in particular ways and with particular codes and labels. Because there are multiple types of users now all accessing the same system, ETS has worked to create explanatory keys within the system, eliminating the need to have prior knowledge in order to decipher the language used to present information. • ETS has maintained grassroots connections with users at the school level, trying to incorporate their feedback about user needs into their ongoing development work whenever possible. According to both users at the school level, and staff at ETS, there is frequent communication between both parties about what school users would like to have access to through the data warehouse, what navigational problems IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 14 are most pressing and how to most effectively organize data at the local level. For example- staff from ETS have consistently attended regular meetings of principals across the district in order to gather input regarding the data these administrators would like access to and the types of reports they would like to run on a regular basis. This feedback is getting incorporated into both the types of queries that will be made widely available through the web interface and in the type of help and training support available to users to improve their effective use of the information. • As part of making data available to stakeholders within the system, the district is providing education around data analysis and interpretation for users. There have been efforts over the last few years from the Department of Strategic Planning within the BCPS to offer data analysis classes to administrators and teachers in order to aid them in their use of the information being mined from the data warehouse. This has led to an increase in session attendance, which was already high, because more of the school-based staff see the direct relevance of the training to their school accountability efforts and are not as challenged by the technological issues that needed to be overcome when the focus was on mastering use of the Brio client. Classroom Practice and Technology Literacy • Teachers now have access to important information about their students prior to the time that their students enter their classrooms at the start of the year, enabling them to tailor their instruction to individual student needs. Though use of the data warehouse by teachers is limited, and information is not uniformly being shared by administrators with teachers, those who are active users of the data warehouse indicate that they are better able to address the needs of their students. These active users say that this is because teachers have access to multiple years of test performance and absentee rates for their incoming students, and therefore can see what sort of performance trajectory and behavioral patterns their students are following. In addition, ETS has made reports on the web available to all school administrators and teachers. Since all schools are not yet Data Warehouse users, some query reports, such as teacher reports with test scores of students based on 2000/2001 class rolls, were made available via the Data Warehouse web site in a widely acceptable format (pdf). Using curricular goals or cluster scores, these reports give teachers an overview of how the students in their current class performed on the FCAT NRT test the previous year. This information allows the teacher to focus on strengths and weaknesses of their students and to redesign lesson plans accordingly. Once the scores for this year are made available, teachers will be able to compare how their students scored both upon entry to their classes and upon leaving at the end of the year. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 15 Evidence of Institutionalization • Schools across the district are using the data warehouse extensively, as are central departments such as Research and ETS. At least two people per school, and in many cases quite a few more, have received training. The data warehouse serves different purposes for different users, but the information that it stores is used at numerous levels within the BCPS system. For example, while a small number of classroom teachers have been Brio trained, these teachers who use the warehouse gather information on students coming into their classrooms in a given year to better prepare to meet these students’ needs. Prior to the introduction of the web-based version of the most relevant queries for teachers, the majority of the schools we interviewed reported that school-level personnel were using the warehouse on a regular basis to print out and pass on information directly relevant to their students. Administrators use the data to gain a better understanding of their student body as a whole and to make curriculum planning and staff training decisions, as well as to set improvement goals that are reasonable and achievable. • The data warehouse is now the main reporting storage system for information on students in the Broward County Public Schools and reports run through the system have replaced reports previously sent by the district to schools through internal mail. Over the course of this program there have been ongoing changes and updates to the information stored within the data warehouse, and the data contained within it has grown considerably from the program's initial specifications. For instance, the number and composition of queries has changed dramatically as the district has evolved a sense of which types of information are the most useful to which people in the school system. Two areas have seen the largest growth in utilization: 1) joint school-based administrator and classroom teacher use of the information to form specific instructional plans based on the needs of the specific students in a given class (in response to increased school accountability mandates) and 2) special program reporting and evaluation. Feeding into this evolutionary process, users have grown more sophisticated and regular structures have been put in place to receive ongoing feedback and suggestions from teachers, administrators and district level about the content of data warehouse and the queries to develop. To make this information as easily accessible to schools as possible, ETS is running queries of test score data for all schools, by teacher, and making these runs available through the Web, and by email, in PDF format. • Schools are going beyond the original design of the data warehouse by adding in local datamarts to merge central and local data at the school level. Schools are using local FileMaker Pro databases in which they store data collected at the classroom and school level and then combine this with data regularly imported from the data warehouse. For example, schools may have locally stored data records containing information such as class grades or teacher observation records, that can be combined with data gathered from the central data warehouse. A movement that began within a single zone of the Broward County school system in order to collect locally derived information on students has spread throughout a number of other IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 16 zones within the district. These databases are not being used in place of the data warehouse, but rather are used in conjunction with them and enable administrators to have access to school-based and warehouse-based data in a single locale. While ETS is not directly supporting these databases, they are working closely with principals across the district to make sure the data warehouse accommodates as much of the data as possible. ETS believes that, as more information is added to the warehouse, the need for these local databases will decrease and the warehouse will be able to address most needs within schools, though there is also an understanding that some school- derived data is too specific to ever include in the warehouse. We see the development of these databases as a sign of institutionalization, however, in that they show that schools themselves are recognizing the importance of easily accessible and manipulative, long term data — both test scores (and other data housed in the warehouse) and more local assessment and behavioral information. Furthermore, these schools are regularly updating their databases with data retrieved from the data warehouse rather than replacing the use of the warehouse, and are feeding information back up to ETS to inform the ongoing development of the data warehouse. They are also increasingly updating the transactional data system, some of which had not been kept up-to-date before, so that the data warehouse contains as much information as possible. • A focus on teachers as potential data warehouse users indicates that administrators are invested in the use of the system and understand the value of the tool in the hands of teachers. The initial target group for use of the data warehouse in Broward was classroom teachers. Quite quickly, however, there was a shift in the user base as schools began sending administrators and technology personnel to Brio trainings, and as ETS worked to include fields and reference codes requested by these administrators within the tool. In part this shift was due to the sharp learning curve associated with the Brio client. The skills and understanding necessary to use Brio are hard to maintain without sustained use, and while teachers need information housed within the warehouse, this need comes at a few critical points in the school year rather than on a daily or even weekly basis. The needs of administrators, guidance counselors, and program directors, however, can support a more continuous level of use. Because these groups have become relatively active users of the data warehouse across the district, steps are now being taken once again to make the warehouse accessible and useful to teachers by developing more simple processes by which to access information. The maturation of Broward's network infrastructure and technology utilization is facilitating this process by making it possible to publish regular queries in quickly and easily accessible electronic formats that are available through the Web and by email, now virtually ubiquitous in schools. The different means by which data can be retrieved from the warehouse, which includes the development and testing of browser plug-ins that run queries, illustrates a sophisticated spectrum of utilization emerging in Broward. This spectrum spans from those who use the full Brio client to create their own, customized queries in order to investigate specific questions (often for a third party), to those who access pre-built queries through an augmented browser, to IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 17 those who regularly update data by repeated use of relevant queries, and finally to those who use data that is part of their planning processes that is pushed out to them by an outside agent. This is a dramatically different picture of data use in a school system than the model it replaces: a research and evaluation department producing reports on paper and using a top-down dissemination model. • The data warehouse has become integrated into the district and users of the tool are largely unaware of the history of the project and its relationship to IBM. When we asked users where the data warehouse came from, respondents either believed that it was entirely a district project, or indicated that that it was through a grant, but were unaware of the details.8 This kind of association between the warehouse and the district is a positive sign that this tool is perceived of as initiated and developed by the district, and not an outside program that the district does not have full ownership over. As further evidence of this institutionalization, the benefits of the data warehouse are extending beyond student achievement and into general school efficiency by storing work contracts and maintenance requests that help individual schools track these logistics. Ongoing Challenges The position of the data warehouse as a district-wide tool, and the context of Broward within a state invested in high-stakes testing as a means for evaluating and grading school performance, makes this solution a key variable in how strategic planning occurs at both the school and district levels. In the next two years, this is likely to spread to the classroom level in a significant number of schools. As such, we see this next phase of evaluation research as an ideal opportunity to investigate how schools plan for and respond to state requirements around school improvement. • One challenge the district faces is understanding specifically how schools use data -- and the data warehouse -- to respond to state and district grading practices and accountability pressures. Every school in Florida is given a grade each year around which their achievement goals are set. Given their initial positive results in using data to respond to achievement challenges, based upon qualitative/anecdotal information we have received, it will be an important part of the ongoing evaluation of the success of this solution to document exactly how schools are using the data warehouse to meet their improvement goals. • The district will want to investigate how data is used at the district level to develop strategic plans for school improvement, and whether or not there has been an increase in interdepartmental communication and planning to address school needs. The data warehouse is not only a tool for school-based users, but is used by departments throughout the system, such as the Research Department to look at school reform efforts. In order to understand the systemic impact of the solution on planning and reform it will be essential to carry out research at multiple levels of 8 System conducted in spring 1999 and spring 2000.. Finding based on interviews with teachers and administrators within the Broward County Public School IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 18 organization within the system. Specifically, we recommend looking at how initiatives are devised and implemented across the district as a whole, and at the kinds of collaborations that take place across departments. • While many school administrators are making use of data extracted from the data warehouse, in some schools this information is still reaching teachers through more traditional channels. While administrators have indicated that having data at their disposal has helped their work processes, they are not all sharing this data in a regular, systematic way with teachers. Because we interviewed only those teachers who use the warehouse, we do not know how teachers are receiving data- driven decisions being made by administrators. There is a risk, then, due to this lack of communication, that teachers will not clearly understand why certain decisions are made and will feel cut off from the flow of information that is occurring at other levels. Related to this, the goal of making student information available directly to classroom teachers has not entirely been met because of the training needed to proficiently use the Brio interface and because administrators do not always believe that teachers are the appropriate targets for user trainings. Though teachers were the initial target audience for use of the data warehouse, relatively few teachers have received Brio training, and many of those who have did not become regular users. Some of the barriers to the use of the system have been lack of access to computers loaded with Brio; problems remembering how to use the skills they were taught during Brio trainings; and early technical problems with the software. The district may want to consider whether it wants to continue training teachers or follow the preference expressed by one administrator to have a point-person. This person would "ensure that the necessary data is distributed to the relevant people" and allow teacher training to be focused on meaningful technology integration into the classroom rather than data warehouse use. • The district will also want to track how the continued development of the data warehouse includes incorporating locally relevant and locally collected data, and how that data is being shared across sites. Though district personnel acknowledge that it is important for schools to be able to store and track data collected locally and not available at the district level, logistically this model is difficult to support. Furthermore, the initial stakeholders who worked to develop and disseminate the school-level databases are primarily administrators, not technical support personnel, and have begun to find their time and energy being taxed as the databases roll-out to more and more schools and they are called upon for their expertise and support. In response to this, one goal of ETS is to incorporate as much of the local information as is feasible and as makes sense (i.e., is relevant across schools) into the data warehouse. It will be important to investigate the relationship between and the trajectory of each of these two efforts in order to determine how successful ETS is in meeting the needs of as many users as possible. • The spread of information about and access to the data warehouse often remains centralized within schools. At times only an Assistant Principal, a Guidance Counselor and/or a technology specialist in any given school has received Brio IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 19 training. The use of the data warehouse, then, becomes dependent on these one or two people, and others in the school only receive the information they determine to be important. As one teacher explained, if a technical person is sent to be trained and decides that the training is not relevant to his or her work, then no one else in the school will learn what is available through the tool. As the district moves to using the Web to deliver specific reports on an automated schedule, and continues to increase the use of browser-based queries, this will be less of a challenge. • Some users at the school level have expressed concern around the use of data to evaluate teacher performance and around the use of test scores as the primary method for assessing student performance. While all of the participants in our case study research were positive about the impact that access to information through the data warehouse has had on their work, there have been a number of concerns expressed around the potential misuse of the warehouse for teacher evaluation and accountability. Specifically, both administrators and teachers have voiced concern that if patterns are found in which teachers repeatedly have students in their classes who score low on standardized tests, whether the district will take punitive measures against those teachers. Similarly, they have pointed out that comparisons of gains in classrooms are often made without taking into account the initial level of the students. Furthermore, the focus on data that the warehouse affords further increases an already high value on test scores as indicators of student learning. To school personnel who understand that there are a number of variables that affect a student's performance on a standardized test, and who have access to multiple forms of student work and assess learning in multiple ways, relying too heavily on test score data alone is of great concern. As one principal said, "Life goes around the FCAT — all pressure is to improve them. Forget about other parts of kids. Guidance doesn't have time to do the counseling they could do before because they are focusing on testing. The only way to help is to hire assistants for them." • The district will want to gauge how the district further increases the data warehouse user base, and whether or not it is successful in its efforts to reach more classroom teachers. ETS has indicated that they are refocusing on how to disseminate information about the data warehouse to teachers, and how to help teachers make sense of that information. It will be important, then, to document these efforts. Specifically, we suggest looking at what is unique about teachers as a user group, what kinds of support they will need to make use of data, and what kinds of support the district can and does give to help meet teacher goals for both their students’ and their own improvement. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 20 Charlotte-Mecklenburg Public Schools District Background Charlotte-Mecklenburg Public Schools is the 23rd largest school system in the nation and the largest in the Carolinas. 1999-2000 Data ü Number of K-12 Students: 99,403 ü Number of Schools: 136 û Elementary 86 û Middle 28 û High 14 û Other 8 ü Number of Teachers 7,017 ü Budget (Recommended 2001-02): $287,289,053 Solution Summary Charlotte-Mecklenburg’s Reinventing Education project focuses on increasing communication about teaching and learning between various stakeholders who are critical to improving student learning. The solution, Wired for Learning, has tools to increase parent-teacher communication, tools for involving parents and the community in the work of the school, and tools for improving internal communications between the faculty of a school, teachers and parents. Initially the project focused on the Governor’s Village cluster of four schools, a special community developed explicitly to foster parent involvement in their child's schooling. The project has now been adopted by the district and was rolled-out to ten additional schools, as well as two community centers to date. Governor’s Village is a unique community, both in its diversity and in its commitment to parent involvement in the education of students. It draws students in roughly equal numbers from three populations: children of people who work in plants and offices located in the surrounding industrial park, children living in an inner-city working class neighborhood, and children of families throughout Charlotte-Mecklenburg who want to participate in the parent-involvement magnet program. Because Governor's Village is a self-contained feeder pattern including elementary, middle and high schools, students can stay within its structure for their K-12 career. Wired for Learning (WFL) with its concentration on promoting communications has evolved a suite of online tools: IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 21  • Homepage Creator provides templates for developing simple Web pages specifically oriented to communicating about what goes on in a teacher's classroom. Users can build and maintain their own pages without knowledge of HTML. • Instructional Planner takes teachers step-by-step through the creation of instructional units, with the capability to search and reference relevant local standards, attach resources, and link to related Web sites. Teachers can choose colleagues to act as online reviewers, and can upload their units to a shared database. • Team Projects provides teachers and students an environment for online collaboration. Teachers can use Team Projects when jointly designing cross-grade or cross-disciplinary activities or units. It also can be used as a place for students to collaborate with students in other classes. • Events at School is a calendar accessible to school personnel, parents, and community members. • Mentors at School facilitates the registration of volunteer mentors. WFL was introduced at Nathaniel Alexander Elementary School (NAES) in 1996-97 and at Vance High School (VHS) in 1997-98, the year each school opened. Technology use was fairly robust at NAES and VHS, due in part to the strong vision of their administration and in part to the fact that many teachers were attracted to these schools by the opportunity to use technology. Teachers reported a high degree of comfort in their own technology use and a number of them used computers and the Internet regularly with their students. Teachers had between two and seven computers in their classrooms, and each school had a media center. Each school provided Wired for Learning workshops for teachers and other personnel, and required teachers to develop and maintain Web pages using Homepage Creator as an initial step in using WFL. There were no formal requirements for the use of the other WFL applications, and teachers at both schools used them to varying degrees. Teachers in several other schools attended Wired for Learning trainings as well. During the two and a half years of this evaluation, computers have become essential to the teaching practices of the majority of teachers in these schools. Email use has become pervasive. For example, the principal of Vance High School had begun issuing memoranda only via email; teachers came to know they had to check their email or risk missing important information. With this base now in place, we have found that communications between parents and teachers is increasing. It includes email exchanges as well as use of the teacher-produced home pages and calendars. In addition to working within the Governor’s Village schools, the Reinventing Education effort sought to increase parent communication. In order to do this, the grant created a technology access point within the community. Computers were installed in the local Double Oaks Community Center, in part to facilitate parents’ electronic communication with school personnel. However, the director of the center reported that parents were not using the technology for that purpose. Changing the center to accommodate a pre-school made it difficult to promote its use as a technology access point for parents. In fact, less than five percent of parents reported having computer access at a community center. The IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 22 WFL coach has made increasing parent awareness and involvement through the Double Oaks Community Center a priority. Plans include outreach through activities at the Double Oaks Community Center and other venues. In addition, the center has assigned a staff person to conduct training and provide support to parents using the center's technology to access WFL. Focus of Evaluation At the outset of the evaluation, the Center for Children and Technology worked with project leadership to develop a set of indicators against which to measure the progress of the initiative. The researchers then developed a plan for collecting data related to the indicators. The indicators for Charlotte-Mecklenburg were: • The degree to which teachers use Wired for Learning in their classroom practice; • The degree to which parents and teachers use Wired for Learning to communicate with one another and the kinds of information they share; • The extent to which schools use Wired for Learning to disseminate information within the school community; • The degree to which Wired for Learning applications are aligned to curricular and organizational goals; and • The degree to which Governor's Village and the Charlotte-Mecklenburg school district display ownership of the Wired for Learning initiative. Current Status In the summer of 2000, the district obtained a grant to support the position of a Wired for Learning "coach" for one year. The WFL coach, along with IBM representatives, has met with the principals of 13 schools, including the original Governor's Village cohort, to establish goals for teachers' use of Wired for Learning over the 2000 - 01 school year. The coach conducted ongoing training sessions on Homepage Creator, and all the teachers have created Web pages using the application. The coach plans to work with parents at the Double Oaks Community Center and other venues, as well. In the fall of 2000, nine additional schools in the district are adopting WFL.9 In the next phase of work project leaders plan to focus on Instructional Planner. The district will further evaluate the initiative in spring 2001 and determine whether to implement it district-wide.10 9 10 Charlotte-Mecklenburg, and Zuni Johnson (date), WFL coach. Per interviews with Ann Clark and Zuni Johnson. Per interviews with Ann Clark (date), principal of Vance High School and coordinator of WFL in IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 23 Evidence of Benefits11 Professional Development • Teachers are using email among themselves to communicate on instruction. According to a fifth grade teacher at Nathaniel Alexander, "We do a lot of emailing one another. That's basically how we communicate with one another instead of tracking down each other on foot."12 The school technology specialists report significant annual increase in the volume of email being sent and received by the teaching staff. This has developed into a basic professional communications tool for much of the staff. Improved Communications • Teachers value and use the WFL tools to increase communication with parents and other teachers. In gathering data regarding the first indicator, the degree to which teachers use Wired for Learning in their classroom practice, we found that the majority of teachers surveyed said that the WFL tools were valuable to their work. Homepage Creator was by far the most frequently used, with Events at School coming in second. All teachers had homepages; over half the survey respondents reported using Homepage Creator at least once a week, and a few teachers were using it extensively with both students and parents. Teachers’ Web pages became more sophisticated, and teachers who used the application frequently found it highly relevant to their work. • Parents use the WFL site and email to get information about school and communicate with teachers. We found a significant amount of evidence to support the second indicator, the degree to which parents and teachers use Wired for Learning to communicate with one another and the kinds of information they share. The number of parents with email accounts increased over the course of the project, as did parents’ degree of comfort using electronic communications tools. Seventy-five to eighty-five percent of parents who responded to a 1999/2000 survey reported having computers at home and over seventy percent said they had an email account and were at least somewhat comfortable using it. 13 Communications from the school, interactions with parent volunteers, and WFL training sessions have helped build awareness amongst the parents that this is a viable communication option. Teachers reported an increase in parent email communication during that time as well. In some cases as many as half of their students’ parents communicate electronically. A number of parents also were using WFL to obtain information about their children's 11 administrators conducted throughout the course of the evaluation. All references to comments and findings are based on interviews conducted with teachers and 12 13 It is important to note that parents who do not have access to computers may be underrepresented in these findings; those parents with access are probably more likely to complete and return a survey about computers and the Internet than are parents who do not have access to the technology. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 24 Telephone interview, March 21, 2000 homework assignments, school activities, and scheduling. One parent said in an interview that she kept up with her child's assignments through the teacher's Web site so that she could plan the family schedule accordingly; when she knew there would be a heavy homework day, she kept other activities to a minimum. The numbers of parents using this resource varied considerably depending on how many parents in a given class had easy access at home or work, on how often the web pages changed for their student’s class, and the amount of homework required. At the high school level, it was usually the students themselves who checked homework assignments, while parents use this feature for students in the middle and upper elementary grades. • Teachers report that email offers several significant advantages as a vehicle for parent-teacher communication, and that email from parents, including parents who did not communicate as much in the past, is increasing. Teachers report that it is more convenient and less time-consuming than other options. It makes them more accessible, because they do not have to be in a particular place at a particular time to interact with parents. They also say that parents are more likely to respond in a timely manner to an email message than to a telephone call or a letter. A ninth grade social studies teacher at Vance High School said, “First year, I had very few emails, and these tended to be from parents of the top students. With these it was often the same message: 'your student is excellent.' The number of email messages has gone up and the variety. Now I communicate with plenty of at-risk students' parents. I do love the positive messages but the students that need a weekly reminder, those are the ones that need the access more than ever. ... There is a much larger cross-section that goes across race, demographics, socio-economic factors. We’re getting the people that you need to really get to."14 Classroom Practice and Technology Literacy • A few teachers use the tools, specifically Team Projects, to augment a shift to project-based learning and further engage parents. A group of fifth-grade teachers used the application for threaded discussion among classes involved in the Scholars Project, and trained parent-assistants to use Team Projects, as well. One fourth-grade class used Team Projects to develop their science fair projects. They posted their ideas on their team site, and invited adult mentors to review them and offer comments. Evidence of Institutionalization We found strong evidence of the last indicator, the degree to which Governor's Village and the Charlotte-Mecklenburg school district display ownership of the Wired for Learning initiative. The administrators, technology coordinators, and instructional facilitators we spoke with are strong advocates of WFL. They see the applications as a powerful set of tools that can support their goals of increasing communication and collaboration within the school community and beyond. Additionally, they have initiated directives to staff and programs to use WFL and provided training and support for its use. 14 Telephone interview, March 8, 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 25 For example, the teacher leader has had meetings with principals in order to set requirements for teachers to update their web pages. • In the summer of 2000, a Wired for Learning coach was hired.15 Resources have been obtained to staff this position, demonstrating the district's commitment to deepening the utilization of the tools. The coach's job is to ensure that participating teachers receive the ongoing training and support that will enable them to use WFL effectively. The coach is also introducing parents to the WFL environment. The WFL coach is likely to be a critical element in the ongoing success of the project. Previously, the project coordinator, Vance High School principal Ann Clark, worked with the administrators and technology coordinators of the other participating schools to set goals and implement professional development activities. As the project has expanded, however, the amount of attention and effort required has increased exponentially. The WFL coach is already demonstrating the ability to implement plans and respond to problems more quickly and efficiently. • The technology infrastructure is in place and is perceived to be reliable. The technology support staff is seen as highly efficient and effective. Vance High School administrators report that they submit fewer requests to the district's central technology support office than other schools in the district, which suggests that Vance technical staff are solving more problems internally. Teachers are more likely to use new applications if they feel they can trust the underlying technology and that they can get help when they need it, and both of these conditions appear to be in place. The human infrastructure that supports the technology is also in place as the roles of the technology coordinators within the schools are shifting from those of troubleshooters. Now there is an increased emphasis on training and integrating technology into the curriculum. It is likely that this trend will facilitate the introduction of Instructional Planner. By the same token, the implementation of Instructional Planner will serve to strengthen the focus on technology integration as more teachers apply their technology use to their classroom practice. • Technology use, especially of WFL, and communication with parents, are both increasing, though slowly. According to teacher and parent surveys, teachers, and to some degree parents, are becoming comfortable with the technology and the Governor’s Village community as a whole is ready to move on to a more comprehensive level of technology integration and use. • All teachers in participating schools have home pages to communicate information on their classrooms to parents. While in the past, few teacher pages were updated regularly, the principals of participating schools are establishing guidelines for updating teacher pages and are offering incentives in the form of continuing education credit.16 Moreover, if parent use of the home pages increases, as is the goal, teacher motivation will likely increase as well. 15 16 Per interviews with Ann Clark and Zuni Johnson. Per interview with Zuni Johnson. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 26 • Plans are in development to implement Instructional Planner in participating schools. The implementation of Instructional Planner stands to accelerate the integration of WFL into curriculum, as well as communication among teachers, as they co-plan and share their instructional units. Ongoing Challenges • The district needs to find ways to inform parents about WFL and encourage their involvement. While parents report an increase in their use of technology, it appears that few parents are visiting teacher Web pages. A significant number of parents said they were not aware that they could get school information through WFL, though nearly half the parents surveyed were interested in WFL training. This situation is frustrating to teachers, who feel that it is not worth the effort to update their pages regularly if their target audience is not going to see them. In addition, though many teachers reported having regular email communication with parents, these parents were sometimes the ones who were more likely to keep in touch in any case. While parent use of technology for communicating with teachers has increased, the district would do well to reach an even wider audience of parents, especially those for whom regular contact has not occurred. Teachers see parent access to technology as a significant barrier to widespread participation in electronic communication.17 While a number of parents reported having access to computers at home or at work, teachers felt that many parents are still without access. (Though it is too early for current outreach efforts through the Double Oaks Community Center to show results, if successful, these efforts will go some distance towards addressing this issue.) • The district needs to encourage teachers to update their homepages on a regular basis and provide training and/or structure for substantive and meaningful homepage content. A content analysis of teachers’ homepages in the spring of 2000 indicated that while most teachers modified their pages from time to time, few updated them frequently. This was largely due to the perception of time required to keep Web pages up-to-date and the relatively low number of parents visiting the same teachers' Web pages. In addition, most pages did not directly provide links for parents to use in exploring the content being taught, provide homework activities, assessment information, or other resources that could make these pages a more integral part of the instructional process. For the most part, the pages were used to inform parents about year-long, general information or specific logistics related to events. Thus many teachers are caught in a conundrum: their pages are not the sort that stimulate frequent visitation, and as a result they don't get the level of visitation needed to push a greater 17 Ibid. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 27 commitment of time to creating and updating the sort of pages that will promote greater use by parents. The foundation is in place for more extensive and strategic use of the pages, and current efforts to establish guidelines for Web page maintenance at individual schools are designed to address these challenges, as are efforts to raise parent awareness and use of WFL. • Similarly, the district must find ways to encourage increased use of other WFL applications. Almost half of the teachers surveyed said they had little or no motivation to use Mentors at School, Instructional Planner, and Team projects. Time constraints coupled with competing (non-technical) planning procedures appeared to be the biggest impediments to integrating WFL into classroom practice more fully. These specific tools have not yet been integrated into the existing planning processes used by the staff, however, the district plans to make Instructional Planner the focus of the next phase of implementation. One of the limiting factors of WFL use has been that within-school communications often do not need to leverage technology since there are plenty of face-to-face opportunities, especially in the Governor's Village schools that have ample teacher planning rooms and frequent staff and committee meetings. Use of email for internal communications has increased substantially but having more geographically distributed schools on the system should help privilege more substantial use of the WFL tools. • To sustain and scale-up the project, the district must identify opportunities to integrate WFL into other initiatives. Projects that are not integrated with other initiatives within the school system are more vulnerable to the effects of staff turnover and changing administrative priorities. Such integration would help to solidify the foundation that the WFL project has established thus far. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 28 Chicago Public Schools District Background Chicago Public Schools is third largest school district in the United States, serving as the home to approximately one out of five Illinois public school students. ü Number of Students: 435,470 (September 2000) û Eighty-six percent of students are from low-income families, more than 90 percent are minority, with 13.8% limited-English-proficient. ü Number of Schools: û Elementary (1-8) û Secondary û Other ü Number of Teachers: ü Operating Budget (Fiscal year 2001): Solution Summary 596 491 92 13 26,348 teachers (1999-2000) $3.5 billion (estimated) The focus of the Chicago Reinventing Education project is professional development for seventh and eighth grade math teachers. The district found that the majority of Chicago Middle School math teachers held elementary school certifications rather than secondary school certifications, limiting their ability to effectively teach secondary level mathematics concepts. The goal of the RE project was to infuse technology into the curriculum for 7th and 8th grade mathematics teachers. Through the use of this technology, teachers would have the ability to enhance their own content knowledge and share their experiences with their peers. IBM’s Learning Village, briefly referred to in Chicago as “Teacher Connect”, is the suite of applications that was adopted and is being used to support the Chicago Reinventing Education project. Chicago Public Schools (CPS) received a $2 million dollar Reinventing Education grant in 1995. As a result of shifts in district leadership, the first Phase of the Chicago Reinventing Education project was not launched until 1997. Phase I began with thirteen participating teachers from six different schools. The project aimed to achieve the following goals: 1) Provide teachers with the training needed to build their math content knowledge in new areas that they were now expected to cover; 2) Provide teachers with curriculum and instruction materials to support teaching and learning in these areas; and 3) Provide seventh- and eighth-grade math teachers with an online environment to foster ongoing professional communication. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 29 Through the grant, each Phase I teacher was given a laptop, four classroom computers, a scanner, a printer, a large screen monitor, and access to a T-1 line for their classroom. The project leadership hired the Math Group at the Education Development Center to design curriculum and conduct teacher training around Geometry, Algebra, and Measurement and Data Analysis — the three major areas of concern in the teaching and learning of mathematics identified by teachers. The Chicago Reinventing Education project was also focused on working collaboratively with schools to ensure that the technology infrastructure needed to support use of the mathematics tools was in place. Phase II of the project began in June 1998 and involved 38 participants from 17 additional schools. These participants were teachers from schools with sufficient technology to support the training and meet the requirements of the Web-based applications. Principals nominated teachers and final decisions were made by IBM and CPS based on each school’s current technology, a letter of commitment from the school’s principal and a resume of the nominated teacher. Selected teachers participated in three two-day training sessions. Each of these training sessions was organized around one of the three areas of concern: Geometry, Algebra and Measurement and Data Analysis. Along with being provided Geometry and Algebra software (GeoShapes and Mystery Rule), the training provided teachers with math curriculum guides and classroom tools, such as pattern blocks, hinge mirrors, and circle protractors. In terms of technology, Phase II teachers received one personal computer, one printer and a Think Pad, to be used at home. All Phase I teachers utilized the “Wired for Learning” web-based application. Through this tool they collaborated with one another around new methodologies and concepts in the teaching and learning of mathematics. In Phase III of the project, the district is successfully leveraging the Reinventing Education work by bringing in new dollars – e.g., the Department of Education’s GEAR UP and the National Science Foundation Chicago Urban Systemic Program (CUSP) grants. As these grants support complementary district efforts, they are scaling up the technologies to new schools by using them to train significant numbers of new teachers and to serve as the communications vehicle for ongoing work. Focus of Evaluation At the outset of the evaluation, the Center for Children and Technology worked with project leadership to develop a set of indicators against which to measure the progress of the initiative. The researchers then developed a plan for collecting data related to the indicators. The indicators for Chicago were: • Changes in the teaching practices of Reinventing teachers with regard to classroom use of technology; • The extent to which teachers were both trained in and used Reinventing Mathematics tools (both hardware and software) in their classrooms; • The extent to which the project was able to facilitate professional development and increase teachers’ skill and comfort level in mathematics content and instruction; IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 30 • How the program was able to scale-up to accommodate new participants, district clusters, and project directions; and • The extent to which the project was able to generate new funding sources to support and maintain the technology infrastructure of the project and the evolution of the project. Current Status Throughout the duration of the Reinventing Education project, it had been the intention of CPS and IBM that the infrastructure built in the planning and first two phases of the project would be sufficient to sustain and grow the project into the future. Significant strides have been made in this direction. Now in Phase III, CPS, with the support of independent educational organizations and additional funding sources, has initiated several scale-up activities that are in various stages of planning and development. Participating teachers are being trained on various aspects of Learning Village, such as Home Page Designer and Team Projects, as well as on the hard copy math tools. Throughout the planning process and Phase I and II, CPS has been deeply involved with two independent organizations: The Chicago Urban Systemic Program and the Department of Learning Technologies. Until now, all of the scale-up activities have been under the direction of IBM. Phase III goals are to get CPS rapidly to a point where they can continue independently. IBM will be able to assist CPS in its scale-up activities of Learning Village through additional assessment, planning and service contracts. Included within the technology environment of the Reinventing Education scale up project are the Wired For Learning applications, such as Instructional Planner, Try Science, and Home Page Designer, and the math tools, GeoShapes and Mystery Rule, created during the first phase of the Chicago project. The ultimate goal is to have Learning Village in all of the approximately 200 district schools that are a part of the Chicago Urban Systemic Program. CPS is currently in the process of taking over control and oversight of all project efforts. CPS will sign a licensing agreement with IBM defining the use of Learning Village within the school district. This agreement will provide licensing for up to 2,500 users, adding 2,000 licenses beyond the 500 that were provided as part of the IBM grant. The 2,500 licenses will cover the planned scale-up activities beyond the currently involved school sites. Presently, the teachers who have participated in the project have been trained on the base applications of Learning Village. These include Home Page Designer, Team Projects and other threaded discussion tools. In addition, the math tools from the original phase of the project have been placed on Instructional Planner. Specific Instructional Planner training is being planned for this summer, though this work was not part of the original grant, thanks to leveraging of the project through a Department of Education Gear up Grant, and a National Science Foundation (NSF) grant called CUSP (Chicago Urban Systemic IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 31 Program). As evidenced in West Virginia, implementation of the Instructional Planner will compliment the Reinventing Education project by providing teachers with a proven means of strengthening their standards-based instruction, as they create lesson plans and share best practices with one another. Evidence of Benefits18 Technology Integration in Core Curriculum Areas • Teachers felt that the tools aided their mathematics instruction in Geometry, Algebra and Measurement and Data Analysis. When their school technology infrastructure supported it, the teachers were using the tools and consider them useful in addressing areas of math where students had previously performed poorly, Some teachers reported that the use of these tools helped them improve their instruction, experiencing greater success in meeting the academic needs of their students. Evidence of Institutionalization • There is a commitment on the part of the district in implementing and expanding the use of Learning Village as a professional development tool. The wavering commitment on the part of the district resulted in stagnant periods, leaving participants uncertain of the project’s future and fracturing efforts to build critical mass around the use of the tools. At the district level there is a renewed commitment to the Reinventing initiatives; project leaders are responsive and enthusiastic when speaking about their projects. Furthermore, as Reinventing Education project is being incorporated into larger professional development initiatives, there are several avenues through which additional funding and support for the project are being secured. The district is continuing with the work begun under the Reinventing Education heading, however, the project goals are now rolled into larger, parallel initiatives, so the RE project no longer has a separate identity. • As part of the GEAR-UP Initiative funded through the Department of Education, Roosevelt University is using the Learning Village tool to train teachers from seven schools.19 Gear Up is a U.S. Department of Education program aimed at improving student performance and preparing them for successful post- secondary education. A governing body comprising of eight universities, the State Board of Education, Chicago Board of Education and Chicago Education Alliance is 18 All references to comments are based on interviews conducted with, and surveys administered to, teachers and CPS administrators. 19 Gear Up is an U.S. Department of Education program aimed at improving student performance and preparing them for successful post-secondary education. A governing body comprising of eight universities, the State Board of Education, Chicago Board of Education and Chicago Education Alliance is leading the grant. The grant structure consists of 6 clusters of CPS schools, each lead by a university partner. Roosevelt University, one of the university partners, has selected IBM’s learning Village as the communication tool for their activities. This is being watched as a pilot by the remaining five clusters of schools. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 32  leading the grant. The grant structure consists of 7 clusters of CPS schools, each lead by a university partner. Roosevelt University, one of the university partners, has selected IBM’s Learning Village as the communication tool for their activities. This is being watched as a pilot by the remaining five clusters. Although GEAR UP is independent of the Reinventing Education Initiative, the use of the Learning Village20 marks the first roll out since the tool was previewed with Phase II teachers. There is also the prospect of rolling out the tool to the other 28 schools participating in GEAR UP. • The IBM e-mentor Program, which pairs IBM staff with students and teachers in one-on-one email dialogues, is using Learning Village as the primary communication tool. This is serving to expand the base of the Reinventing Education grant. Through Learning Village, teachers are able to monitor the mentor- mentee dialogues, and students are not required to have independent email addresses. The e-mentor program currently involves approximately 700 people: Teachers and students from eight schools and IBM employee mentors. There was an official kick- off before Thanksgiving where students and mentors met and began their online relationship. After the initial, getting-aquatinted online discussions, teachers have been monitoring the dialogues and providing curriculum content discussion topics. • IBM and CPS, in conjunction with Roosevelt University, are developing a Professional Development Certification Tract, which will satisfy state and CPS requirements for credit toward teacher certification, re-certification, and graduation. The curriculum for this professional development initiative is being developed on Learning Village and the CPS math modules. This course will focus on integrating the use of technology into instruction and student learning and will be piloted this summer, with general availability in the fall. • CPS is training additional teachers using Learning Village applications. CPS developed a master list of 139 schools and prioritized it into three “Waves” of training in Learning Village applications with a focus on Home Page Designer and Instructional Planner as well as the Chicago Reinventing Project’s geometry and algebra software. This training will be funded through a combination of district funds, the NSF - CUSP Grant, and the Department of Education GEAR UP Grant. • CPS has received additional grants to further their professional development program. This past summer CPS was officially awarded a NSF grant for the duration of three to five years. The goals of the NSF grant are in alignment with the Reinventing Education Initiative and the Learning Village software is being incorporated into the work plan of the proposal. • Teach for America, which is working closely with CPS CEO Paul Vallas to attract alternate path teachers to the CPS system, is interested in using Learning Village as a collaboration tool for their teachers. Teach for America places its 20 Learning” when describing the suite of online tools. Projects participants and partners in Chicago often use “Learning Village” rather than “Wired for IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 33 teachers across the district and is excited to use this solution to sustain teamwork and communication of each “class” despite the physical separation of the members. • CPS has taken over the housing and maintenance of the server. The project team has grappled with determining where to house the project server so it can be used reliably, given the problems that plagued the pilot and the Phase I work. After the completion of the pilot phase of the grant, transitional steps took place to allow CPS to take over responsibility of the support and maintenance of this hardware. As Phase II began and additional equipment was ordered for the project, the installation and support of this equipment became the responsibility of CPS. With additional support, they have installed a new server with the Learning Village software along with the math tools developed earlier and used in the pilot Phase and Phase I. • Teacher Continuing Education Credit or University Credit for participating in the online math modules. It is an objective for teachers to be able to earn their professional development credits from the district through the use of Reinventing Education online tools. This objective serves as another example of how Learning Village has been incorporated into the overall structure of CPS. Ongoing Challenges • CPS will want to consider specific follow-up and support mechanisms for teachers who have been trained in Reinventing Education materials. The lack of follow-up has currently left teachers uncertain about how to integrate the Reinventing mathematics tools into their daily practice and stalled more comprehensive use with faculty members in their schools. A more specific and comprehensive idea of how trained teachers are to disseminate their knowledge to colleagues and what integrated use of the tools in the classroom looks like will benefit the project. • There was a discrepancy between the abundance of technology in the training sessions - and the tools teachers are learning to use in their practice - and the amount of technology they actually have in their classrooms. Over the course of our evaluation, one of the obstacles we observed is that technology’s role in aiding the professional development goals of the district has not reached a level of impact where teachers are able to transfer what they experienced in training into their classroom practice. While teachers found the training worthwhile, they have encountered barriers that inhibited them from being able to use the Web-based resources in their classroom settings. The professional development training will have a more lasting and meaningful professional development experience when workshops are coordinated with technology implementation within participants’ individual schools. To date, the lack of a reliable Internet connection is one of the reasons trained teachers use the offline resources (hinge mirror, protractor etc) more frequently than the online ones. Upon the completion of the second phase of training, IBM previewed the Learning Village especially customized for Chicago teachers. With the renewed availability of the Learning Village site and the rapid increase of connectivity due to IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 34 e-rate funding, the question of whether teachers access the Web-based resources they are provided with is a question worth pursuing. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 35 School District of Philadelphia District Background The School District of Philadelphia is the seventh largest in the nation by enrollment. 2000-01 Data ü Number of Students: 208,170 (including early childhood programs) û Over 80 percent of students are minority. ü Number of Schools: û Elementary û Middle û Neighborhood/Magnet High û Other 264 175 (including one K-12) 42 32 15 ü Number of Classroom Teachers/Department Heads: 11,097 ü Budget: $2.074 billion Solution Summary The Philadelphia School District, through the Reinventing Education initiative, designed a professional development model to improve teaching practice and enhance professional communication among teachers. One emphasis of this initiative, known as Continuous Practice Improvement (CPI), is the effective integration of technology into the classroom. The structure of the CPI model is unique as it is based on a laboratory approach to professional development, rather than on a more traditional workshop model. Through this model, participating teachers spend two to four days observing and working closely with experienced host teachers in their classrooms. Participating principals and project leadership initially selected these host, or “resident,” teachers based on the interest, experience and skill they have demonstrated in creatively integrating technology into a project-based curriculum. In order to minimize the loss of valuable instruction time for students, trained substitute teachers spend the day prior to the visitation in the visiting teacher’s classroom, familiarizing themselves with routines, procedures and classroom content. The ultimate purpose of the model is to improve teaching practice; technology integration is merely a means to get teachers reflecting on how they have approached their classrooms in the past. CPI achieves this goal through a combination of classroom rotations, technology training, and by fostering an on-going, collaborative relationship between visiting and resident teachers. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 36 Another component of the RE project in Philadelphia is the CPI tool, an application within the online environment of Learning Village21 designed to facilitate on-going communication and collaboration between resident and visiting teachers. In addition to providing a space where teachers can communicate with one another, this tool enables them to openly reflect on their experiences through the use of online journals. These journal entries are available to be read, commented on, and benefited from, by all project participants. Finally, the Learning Village environment assists in the logistics and overall management of the CPI project. Yet another aspect within the Philadelphia RE work was the researcher-teacher partnership in elementary classrooms that focused on using cutting-edge speech recognition software to develop tools that assist in developing reading proficiency among early readers. This work was pioneered at the early Philadelphia school sites. Later the resulting program, Watch Me! Read was implemented in Houston as the central component in their Reinventing Education II partnership. An independent evaluation study by Vanderbilt University of the impact of the program there reported significant improvements for both English-speaking and bi-lingual first graders in both word recognition and comprehension. The study also reported marked increases in student motivation to read using the program. The Philadelphia School District is organized into twenty-two geographic clusters. (A cluster is comprised of several elementary and middle schools and one high school, all within the same region.) During the 1997-1998 school year, the project was piloted in the Olney school cluster. Twenty-four teachers from two participating schools took part in the project during the first year. The project has now rolled out to a total of nine school district clusters with the number of participating teachers close to 300. Through the Technology Challenge Literacy Fund (TCLF) grant, four more clusters will be added to the original Olney cluster and the 8 CPI expansions, which will introduce Instructional Planner and use CPI beginning in the summer of 2001.) Focus of Evaluation At the outset of the evaluation, the Center for Children and Technology worked with project leadership to develop a set of indicators against which to measure the progress of the initiative. The researchers then developed a plan for collecting data related to the indicators. The indicators for Philadelphia were: • The extent to which professional communication among participating teachers increased; • An increased sense of teacher professionalism resulting from reduced feelings of isolation and greater opportunities to collaborate with peers; • The degree of comfort and support teachers felt integrating technology into classroom practice; 21 describe the online suite of tools rather than “Wired for Learning.” As is the case in Chicago and other Reinventing Education grant sites, “Learning Village” is used to IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 37 • Changes in teaching practice with regard to classroom use of technology, specifically use of supplemental online lessons and new technology to enhance content instruction; • The extent to which the project was able to facilitate professional development through the development of resident/visiting teacher partnerships; • The extent of institutionalization of the CPI professional development model in the district as a whole; and • How the program was able to form new partnerships and interactions between other community and district reform programs. Current Status The partnership between IBM and the Philadelphia School District (SDP) has evolved into a larger scale project extending well beyond the original Olney School Cluster. There are currently nine of twenty-two district clusters participating in CPI. The number of participating visiting teachers has risen to 300. There are presently 35 resident teachers, each of whom participated in an intensive training institute in summer 2000. CPI also has been expanded to support a principal professional development model called Principals Actively Involved in Renewal or PAIR. This sizable scale-up indicates an increased and consistent level of support from administration and district personnel. In addition, the district is using state funding to implement Learning Village’s Instructional Planner. This will complement the CPI project, since teachers will now be able to use this application to further strengthen their standards-based instruction. An additional 75 teachers will be trained over the summer on Learning Village’s Instructional Planner. Another important addition to the project is the Instructional Technology Point Person (ITPP). The district created the ITPP in each of the participating clusters to facilitate the expansion of the program, and to provide additional technology support for visiting and resident teachers. The ITPP is responsible for coordination of the CPI model as well as post-rotation follow-up with visiting teachers. The ITPP offers ongoing individualized support to visiting teachers in their effort to integrate new technology into their classrooms. Their duties are varied and range from troubleshooting, observing, modeling to peer coaching as different needs and challenges arise. In a conscious effort to support long-lasting change in teaching practice, the ITPP follow-up schedule is spread over four months after the initial classroom rotations occur. The ITPP visits teachers in their classrooms once a week for the first month, once a month for the next two months, and finally, a last visit four months after the initial visitation. The project has undergone multiple staffing changes across all administrative levels. When the project was first piloted in the district, the shift in district superintendent caused some changes in the project’s initial focus. On the IBM side, personnel shifts impeded the consistency of the project and temporarily offset the high level of commitment established by those leaving the project. In addition, when personnel on the cluster level changed, it took time for IBM to reintroduce the tool to the new staff and persuade them of its value. District and IBM personnel report that this period of IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 38 uncertainty seems to be over now that the Philadelphia Education Fund (PEF) is more involved. New leaders have taken on responsibility for the initiative’s current implementation and future growth. The PEF provided leadership and assistance during the Transition Phase, and the School District of Philadelphia (SDP) Learning Technology Support Group (LTSP) is responsible for the current success with expansion. On the level of project management, personnel are invested in the success of the project making the project less susceptible to any future shifts in leadership on the district level. Additionally, as the scope of the project has spread, a comprehensive structure for training new resident teachers has developed. During the current year, new resident teachers attended a two-week summer technology institute, divided between classroom visitations and specific technology training. New resident teachers spent time rotating through classrooms of veteran resident teachers in much the same way as visiting teachers will eventually work with them in their upcoming rotations. They also participated in technology and teaching workshops where they were introduced to specific student production tools, uses of the Internet and search strategies, and specific applications within Learning Village. In addition to this, all 2000-2001 CPI teachers participate in 15 hours of Saturday workshops as well as 15 hours of online work. In a further effort to effect change, all CPI participants, both visiting and resident teachers, are required to reflect on their practice, the training, and their learning process, through the use of online journals. These journals are open to participants to read, respond to and benefit from. Part of the requirement for the program was also to submit online, pre- and post-rotation surveys to help evaluators and developers identify the program’s specific areas of strength and need. Participating resident teachers received $3000 worth of technology enhancements for their classroom and visiting teachers received a computer, printer, and a student age-appropriate software bundle Link to Learn state and Technology Literacy Challenge federal grants, to purchase state-of-the-art computers and Internet access for their classrooms. They also earned district professional development credits for participation in the CPI program. Professional development for school administrators is another important component to the Philadelphia Reinventing Education initiative. The CPI Model of professional development for teachers made a significant enough impression on district and administrative personnel, that a partnership between the school district, IBM and PEF designed a parallel model of principal professional development called Principals Actively Involved in Renewal (PAIR). In this model, principals opt for either a one- or three-day rotation in a school that exhibits exemplary practice around areas such as Balanced Literacy, reduced class size, or technology integration. PAIR is also a tool on Learning Village, which supports the content and overall structure of these rotations, just as the CPI tool supports the CPI professional development model for teachers. There are currently 70 principals enrolled who will begin rotations in the Spring of 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 39 Evidence of Benefits22 Professional Development • Teachers felt that CPI was an effective professional development experience. In both survey and journal responses, teachers expressed that the professional development they received through CPI strengthened their classroom practice. According to teachers, this experience was very effective in exposing them to ways to use computers in the classroom. Most teachers were enthusiastic in their praise of the program and felt that they would recommend it to their colleagues. In all but one of the surveys from the fall and summer of 2000, teachers said that the CPI Model met their needs. Most teachers felt that the CPI Model: • Enabled them to increase student motivation and participation in their own learning; • introduced them to new technologies; • introduced them to new teaching practices; • helped reduce their professional isolation through e-mail or collaboration with others; and • helped them to introduce resources or materials that are not available in textbooks or in the library and satisfied district curricular and/or professional development requirements. The only area that some teachers felt CPI did not influence was helping students feel more a part of their community.23 • Teachers have a sense of ownership over the CPI model. One of the strengths of the CPI model is that teachers have been involved from the inception of the program in its shape and design. Teachers have been encouraged to share their ideas and invited to respond to issues that the model has confronted. Through this involvement, not only do teachers have a deep understanding of the model; they feel a sense of ownership and responsibility toward the success of the program. • The CPI model has produced an emerging cadre of teacher leaders. One benefit of participating in CPI has been that teachers are taking on added responsibilities that are expanding their professional orientation beyond their individual classrooms. As the project has evolved, a few of the teacher participants have taken on leadership roles and are now involved in managing the project. Many of the resident teachers who began as visiting teachers were able to hone their skills and are now willing and able to host incoming participants. Additionally, some of the project leaders who 22 All references to comments are based on interviews conducted with, and surveys administered to, teachers and administrators, and an analysis of comments teachers posted to the discussion and journal areas of the online environment. 23 This is based on 29 post-rotation surveys from the summer of 2000 and 10 post surveys from fall 2000. The first question is a yes or no response asking whether or not the program met teachers needs, and the second question includes forced-choice responses where teachers responded either “yes” or “no” to a list of possible outcomes of the training. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 40  began in positions of lesser responsibility are now providing technology training and are administering the project. Improved Communications • The depth and nature of professional communication among teachers has increased. This is an important change because during early project implementation phases, teachers did not have a strong incentive for using the online CPI tool to communicate with one another. The project was piloted in two schools in the Olney school cluster, Clara Barton Elementary School and Central East Middle School. The teachers at these schools were involved in assisting project leaders expand the initiative to other teachers within their schools, as well as to teachers from other schools in their district cluster. When the initiative was in its pilot phase and first year of implementation, it became apparent to project leaders that most teachers were not using the tool in order to communicate with each other. Because most of the teachers worked in the same schools, there was no real need for a tool to facilitate communication electronically; teachers would simply consult with one another in the hallways and teachers’ lounge. Furthermore, teachers veered away from using the tool because of the frustration they experienced when they encountered glitches with the tool. As the project has expanded, and more teachers have become involved from a wider geographical area, the depth and nature of professional communication has evolved. In the 2000-2001 academic year the program rolled out to schools from eight clusters outside the original Olney cluster (In 2001-02, four more clusters began implementing the program). The number of visiting teachers grew to 300, with 25 new resident teachers and an ITPP person from each district cluster. In surveys and journal entries, teachers have made numerous comments about communicating with their partner teachers via email, the telephone, and in person. In a fall and summer post-rotation survey, numerous teachers mentioned email as a way in which their professional communication changed as a result of the CPI process.24 As one participating teacher reported, “I now have an email address and can communicate more with my colleagues. I can hold conversations about technology and understand and share information. My interaction has already changed.”25 Many teachers also commented that the nature of their collaboration had changed. They felt as if they were now working more collaboratively and supportively with their peers around the area of technology. “We have already become very collaborative — sharing good ideas and troubleshooting for each other.”26 Classroom Practice and Technology Literacy 24 This is from an open-answer question asking how teachers believe that their professional communication will change. Five out of 10 teachers from the fall survey specifically mentioned e-mail as a way their communication would change, and from the summer group of 23 surveys, seven respondents specifically mentioned email. 25 26 Teacher response from a survey completed fall 2000. Response from teacher survey in fall 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 41 • Teachers feel empowered to share their computer knowledge with colleagues, that their communication is more collaborative, and that they would recommend the CPI Model to others. Through survey responses and journal entries, teachers expressed that while there was still much to learn, they were excited about what technology was available, and their new skills and knowledge. Teachers commented on being excited to return to school to share what they had learned with other teachers at their schools. When asked if they would recommend the CPI model to friends the vast majority responded enthusiastically. One teacher wrote, “Yes, this was a great program. I enjoyed seeing how my colleagues are using technology in their classroom and different classroom management styles. I would recommend it to any of my colleagues who are interested in learning about technology, team building activities, and how to incorporate activities which support different learning styles and strengths.” Technology Integration in Core Curriculum Areas • Teachers have become increasingly comfortable with integrating technology into their daily instruction. While the CPI model is not limited to helping teachers integrate technology into their classroom practice, this component of the project has been the primary focus of most rotations. When the project began, resident teachers were selected because their classrooms were equipped with five or six computers. Visiting teachers were interested in gaining a deeper understanding of technology and in developing strategies for bringing technology into their own classrooms. According to surveys from the fall and summer cohort of teachers, virtually all teachers who answered the question responded that the CPI Model had introduced them to new technology and to new teaching practices.27 Journal entries created by visiting teachers show evidence of insights into other instructional practices and issues, the emphasis of most journal entries is integration of technology into daily instruction. Visiting teachers report being more comfortable using technology and feel technology use may enhance their professional practice. As one teacher stated in a survey response, “I am confident that my classroom practice and management style will change drastically for the good of the students. My participating in CPI has shown me how I can use the computer all day in the classroom.”28 • The development of PAIR demonstrates the district’s belief in CPI as an exemplary professional development model. PAIR is a model of professional development that enables principals to interact and learn with other principals in school settings. Through PAIR, an offshoot of CPI, principals have the opportunity to interact professionally with other principals in their school environments. Typically administrators meet in centralized locations and conference rooms. PAIR enables 27 introduced to new technologies and new teaching practices as a result of CPI. 28 Teacher response to a post rotation survey completed November 2000. One hundred percent of respondents from the fall and summer responded yes to the choices of being IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 42 principals to engage in a hands-on interactive exchange with principals of other schools that model programs they desire to learn more about. These models include Blue Ribbon Schools, Balanced Literacy Schools, schools with reduced class sizes and schools that have integrated technology into the fabric of their instruction. The three-day rotation includes a day of active reflection where principals can analyze and discuss their rotation experience. Like teachers, principals will be expected to keep journals and post their entries online so that other administrators may read and respond to their thoughts. Evidence of Institutionalization • The district is further developing an infrastructure to support continuation of the project after the funding from IBM is complete. The district has incorporated the CPI model into its model for professional development. They have developed a highly structured training program and a training manual for resident teachers. Another indication of institutionalization is that teachers receive district professional development credits for participating in the project. In addition to this, there have been new district positions developed in order to support the project. • Current project leadership has developed a structure to provide ongoing technical support for visiting and resident teachers. Technical problems were a major area of frustration and concern during the pilot year of the program. Teachers were hampered in their attempts to integrate computers into their practice by facing applications that were very slow, or were not functioning at all. While technical challenges still exist, the position of the technical support person (ITPP) has benefited both resident and visiting teachers. One teacher said, “There were people (facilitators) there to help on the phone and computer. They were around day and night for assistance with tech problems and/or other problems that arose. They were knowledgeable and friendly.”29 In addition, the summer training institute raised the technology skill level of many of the resident teachers, making for a more productive experience. Many visiting teachers commented about the great ideas and skill they observed during their rotations. Additionally, there is a Coordinator at PEF who oversees the principal component of CPI. • As a result of the project, the district formed a strong partnership with an outside educational organization, which, in turn, is helping expand the CPI model. The district’s partnership with the Philadelphia Education Fund (PEF), an intermediary non-profit organization, occurred at a significant time when the project leadership was beginning to contemplate scale-up beyond the Olney cluster. At this time, PEF played a crucial role assisting the site in seeking funding sources and revising the model to accommodate the other clusters in the district. The district’s relationship with PEF signaled to teachers that there was a long-term, serious commitment to the CPI model. Furthermore, by relying on the expertise of a well- 29 Response to survey question completed fall 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 43 established community-based organization, the district became less dependent on IBM in sustaining the project. • The district has developed PAIR by adapting the CPI model to include principal professional development. Since the CPI model has been a successful model of professional development for teachers, PEF has worked with the district to replicate such a model for principal professional development. The Learning Village tool has been revised to include PAIR, the principal component equivalent to the CPI tool for teachers. Approximately 35 principals participated in the project the first year of its development. Unfortunately, personnel changes and server problems impeded the progress of the project. This year the situation is more stable. Administrators were introduced to the Learning Village site and PAIR application this summer. From this initial exposure, 70 principals began their rotations in the Spring of 2000. • The district has taken ownership of the server. The upkeep of the server and where it is housed indicates ownership of the project. As of August 2000 the server has been moved from IBM and is now being housed by the district. At first IBM still maintained the server in the case of technical problems, however the district now has full responsibility for it, including providing all level one technology support. • School and district level leaders have demonstrated a strong commitment to CPI by securing additional funds and allocating existing funds to support the project. The CPI model has been supported financially beyond the original IBM grant, at the classroom, school and district levels. Participating teachers were compensated for their time by earning an hourly staff development rate of $22.59 for taking part in the summer training program. The district also received the Link to Learn state grant and the Technology Literacy Challenge federal grant allowing them to provide $3000.00 for new resident teachers and $1500.00 for visiting teachers to purchase state of the art computer equipment for their classrooms.30 This money for classroom technology provided major incentive for teachers to participate in CPI, and also gave them the tools to immediately begin modeling practices and techniques that they learned through their rotations and training. On the school level, project principals in the Olney Cluster used their school budgets to purchase computers for classroom teachers to allow them to reap the benefits of CPI. The CPI model is based on rotations that require visiting teachers to leave their classrooms and spend three-four days in experienced colleagues’ classrooms. The funding of substitute teachers by the district, and the willingness of principals to support having their teachers out of the building for several days at a time, demonstrates a strong commitment to the project. • The district is using state funds to implement Learning Village’s Instructional Planner and other functions. An additional 75 teachers will receive training on the Instructional Planner during the summer of 2001. This will ensure that CPI expansion and the teaching benefits of Instructional Planner are fully integrated. 30 The state grant is called the Link to Learn grant, and provided support from June to December. The federal grant is called Technology literacy Challenge Grant. Beyond this support, the cost is covered by funds from the district. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 44  Ongoing Challenges • As the project continues to expand, the district needs to ensure that the CPI training is carefully planned to meet the needs of teachers. Some teachers expressed concerns in the surveys and journals that more thought should have gone into the planning and scheduling of the rotations. Some teachers expressed concern that they were not matched with teachers who were teaching the same grade level, although when they asked to switch classrooms, their requests were often accommodated. Others felt that the time out of the classroom was too long, and that their students lost valuable instruction time. Others felt that the computer training was either too basic or too advanced, hence did not match their technology needs. While these comments were segregated to a small number of teachers, they do reflect concerns that could be frustrated further as the CPI model expands. SDP is taking these issues into account, however, and making adjustments to their current and future planing. • The district needs to ensure a steady supply of replacement teachers to substitute in the classroom of visiting teachers. A key component of the program is the availability of quality replacement teachers who cover for visiting teachers during their rotations. Given the expansion of the CPI model, the need for substitute teachers has increased. The district has made efforts to respond by using veteran, retired teachers who teachers knew and trusted. According to the district project manager, while finding substitutes is difficult, it is not an insurmountable problem. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 45 San Jose Unified School District District Background San Jose Unified School District is a large urban school district in the heart of the Silicon Valley and is the eleventh largest urban school district in California. ü Number of Students: More than 32,000 û Almost 70 percent of students are minority. ü Number of Schools: 45 û Elementary 31 û Middle 7 û High 7 ü Number of Teachers: 1,800 ü Budget: $211,126,543 Solution Summary The IBM Reinventing Education grant helped support a comprehensive project to reform professional development in the San Jose Unified School District (SJUSD). In the fall of 1996, collaboration between SJUSD, San Jose State University, IBM and the National Center for Accelerated Schools in Stanford focused on two main initiatives. These were the development of Professional Development Schools and the creation of a model clinical summer school professional development program called Collaborative Learning Accelerated Summer School (CLASS.) The Reinventing Education grant and its project team focused on three distinct components of these initiatives: • Facilitating teachers' action research projects at the Professional Development Schools; • Supporting the development of the Collaborative Learning Accelerated Summer School (CLASS); and • Developing a portfolio tool and associated applications on Learning Village to support professional development initiatives at the district. Focus of Evaluation The evaluation focused on these three project components and developed indicators for each. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 46  Action Research The CCT team conducted interviews with 15 of the 35 total participants including, teachers, school and district administrators and the Reinventing Education project team. The interviews were designed to collect information in five main areas: • Participating teachers’ successful completion of action research projects; • How teachers developed action research Projects that focused on their own technology use as well as their students’ use of technology in the classroom; • Changes in teacher practice as a result of their participating in the project; • The extent to which teachers integrated technology into their classroom practice; and • Indicators showing successful scale-up and district ownership. Specifically, the evaluation investigated how action research was integrated into other professional development programs in the district. Summer Clinical Model There were 62 teachers who participated in the 2000 Summer Clinical Model. The second phase of the CCT evaluation contributed and extended the efforts of an evaluation team led by Dr. Bot Curley, University of San Francisco professor. The focus of this component of the evaluation was to determine: • How teachers’ experiences in the summer clinical model were different than in other professional development programs; • How teachers expected that this experience would influence their teaching practice; • How responsive the project was to participant feedback, and how that contributed to the programs ability to change over time; and • District ownership of the CLASS program. Specifically, how the summer clinical model was integrated into other professional development programs in the district. The Portfolio Tool This tool was developed to support teacher reflection and allow teachers to assess their own progress as well as their students' learning. In addition to supporting the summer clinical model, action research, and other district professional development programs, it is designed to support teachers as they fulfill requirements for national certification. The evaluation collected portfolios that teachers completed at the end of the summer clinical model. This content analysis focused on the following: • Participating teachers’ successful completion of a portfolio using the tool; and • The differences between portfolios that were completed using the tool and those that did not utilize the tool. In addition teachers and district administrators were interviewed to determine the following: • Teachers perceptions of the usefulness of the tool; and • District plans to integrate the tool into existing professional development programs. In addition to integrating the tool into the summer clinical model, the project team decided, in fall 1999, to conduct a larger pilot with the Beginning Teachers Support and Assessment program (BTSA). The evaluation distributed feedback forms to all BTSA IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 47  team members who used the tool to determine how it supported and facilitated the relationship between BTSA team members. Current Status The project director for Reinventing Education, who has helped shepherd and design the project from the very beginning, has been given more responsibility for all special programs in professional development. She is incorporating the clinical model into all offerings for pre-service and first-year teachers. The portfolio tool also will be a component of these programs to ensure that all new teachers in SJUSD reflect and document their professional development. In addition, the tool will be fully integrated into the BTSA program. The summer clinical model may be expanded in summer 2001 to include additional elementary and middle schools. (The exact number is yet to be determined.) The project also will include more university partners in order to enlarge the pool of pre-service teachers. The project’s main goal is to provide professional development for experienced teachers while having the majority of pre-service and first-year teachers participate in the summer clinical model. The district is purchasing more licenses for Learning Village to support additional programs in the district. Liaisons from each of the district’s eight high schools, as well as the district’s primary technology resource teacher, have been identified and trained on the various Learning Village applications. One of these is a district e-mentoring program for middle and high school students that IBM employees are participating in as volunteer mentors, as part of a larger corporate initiative. Evidence of Benefits Action Research Professional Development • Teachers are now required to outline an action research plan as part of their requirements for the summer program. The IBM Reinventing project team began working with a group of teachers that were participating both in the first clinical summer program in 1998 and were teachers in the three Professional Development Schools in the San Jose district. The project team held workshops throughout the summer and during the following school year. The program was ambitious. Teachers are expected to conduct research — a difficult task since most have had no previous formal training — while learning new technology skills, integrating these skills into their classroom practice and teaching these skills to their students. To support them, teachers were invited to attend several workshops and were given ongoing technology support. Some of the teachers participating from Professional Development Schools IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 48 that had partnered with a university were able to embed this project work within a formal course on action research that was offered by the university. Although many teachers were successful in their action research projects, some teachers struggled with integrating technology into their classroom practice while simultaneously conducting action research projects. After this first year, the project team refocused its efforts on the summer clinical model, embedding action research into the program, which has proven to be a more successful design. This is because teachers participated in a five-week summer school that integrated action research into an overall program that focused on helping teachers become more reflective about their professional practices. During this summer program teachers were formally introduced to the goals and processes of action research and this enabled teachers to better understand how to embed action research into their everyday classroom practice. Classroom Practice and Technology Literacy • Teachers that conducted successful action research projects believed their teaching practice has significantly improved as a result of their projects. A common trait among the successful projects is that the focus was not solely on technology. Instead, teachers used technology to support innovative projects and classroom practices. Another factor that contributed to successful projects was that these teachers understood the process of collecting and interpreting data. For example, one participant conducted her action research project around staff development. Her main focus was helping teachers integrate technology into fourth- grade history projects. She worked with teachers to create rubrics that incorporated the California content and technology standards and specifically designed projects to target those areas. The action research was around the use of this new rubric and the integration of technology into an already existing curriculum. The project also has had an impact on her new role as a school administrator. She explained that her school is considering block scheduling and she is using the training she received through action research to assess the effects of this new initiative at her school. The Summer Clinical Model Professional Development • Teachers were provided time and support in the spring prior to the summer schools to develop plans for an integrated, thematic instructional program that focused on topics and subjects that are district priorities. All participants, who are teachers new to the district, have joined the summer clinical model for two of the four weeks that it operated (as of 2001, all of the team members are involved in 6 weeks of this summer training, in order to clear their credential requirement for student teaching). During this time each participant was assigned to a classroom team and has worked cooperatively with the team, which was headed by the classroom teacher, to IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 49 plan instruction, evaluate activities and student needs, and provide instruction to students. In addition, student teachers from three universities participated in the summer 2000 clinical model. • Overall, teachers were extremely positive about their experience in the summer clinical model and found it offered experiences unavailable in other professional development programs.31 Each teacher reported that the summer clinical model exceeded his or her expectations. They all reported that the summer clinical model offered a level of reflection, collaboration and professionalism that they had not experienced in other professional development programs. One teacher reported, "It was great, it exceeded my expectations. It was the hardest I've ever worked and it was well worth it. I came out with a better perspective on education."32 The majority of teachers reported that the model provided a “very valuable” professional development experience.33 Notably, 70% of teachers responded that the clinical provided opportunities for them to gain teaming skills, curriculum planning with other teachers and a structured reflection that they plan to implement in their classrooms. Eighty percent of the respondents’ thought that the most important aspect of the model was that it has “used collaboration as a vehicle for professional development.”34 • Teachers participating in the summer clinical model felt validated as professionals, and in most cases have taken on more responsibilities in their school and the district. "It certainly validated me as a teacher. I feel more empowered to be a BTSA mentor." One classroom teacher that has been actively involved for three years and is now an administrator in the project said, "This experience was fabulous. I had a kindergarten teacher in my school participate. She has been teaching for 29 years and she said that she had been looking for this her whole career. This is another teacher that is now reinvested and reinvigorated because of the model. Also, the parents loved it and wanted to know why regular school wasn't like this." According to this administrator, the enthusiasm of this parent is due to the team teaching and the use of more innovative curriculum during the summer clinical model. • Teachers who had been involved for two or more years felt that the major strength of the program is its ability to build on strengths and improve over time. Classroom teachers reported that project administrators listened to their feedback and improved the program based on that feedback. For example, one teacher reported that the collaboration among her beginning teacher, her student teacher and herself, was an essential component for the success of the summer clinical model. In the first year, however, the beginning teacher was not involved in the curriculum planning and was not assigned to the classroom until one week after the program started. Classroom teachers believed that the participating teacher and the student 31 The evaluation team conducted interviews with a sample of teachers and distributed surveys to teacher teams after the 1999 and 2000 clinical model summer programs. 32 33 34 Interviews conducted with participating teachers, summers 1999 and 2000. Sixty-two teachers completed a survey after completing the summer program. Surveys distributed to participating teachers, summer, 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 50 teacher had to be involved in the curriculum planning in order to be fully active participants. By the second and third year, changes had been made so that this level of collaboration between the teachers was built in. One classroom teacher said, "The first year my partner did not know the curriculum and she wasn't there until one week into the program. So, in the first year the participant came in too late to really collaborate on the curriculum planning. Then, the second year they came in during the first week, and by the third year they started before and they had more buy-in and more involvement. This past year they were really ready to get involved and not be a bystander." Another teacher said, "I taught all three years, from the beginning. The evolution from the first year to year three has really given us the model that we wanted. Everything was in place. The participants, the student teacher piece, the thematic teaching, the teachers could see the vertical curriculum. It really worked."35 The design of the summer 2000 clinical summer program was largely influenced by feedback from teachers who participated in the program in previous summers. For example, several teachers reported that the district needed to focus professional development around bilingual education. Therefore, the project team developed a bilingual summer clinical model at Olinder elementary school, which received financial support from the district for the following year. • Each teacher has taken what he/she learned during the summer and integrated components of the clinical model into his or her classroom during the school year. All teachers reported using the model’s formal reflective structure, “Observation, Reflection, Insight, Decision” (ORID), with their grade level teams and one teacher has even used it with her students. This has been particularly true at one school where there were four teachers who participated in the summer clinical model for two or more years, and one teacher who is now the acting principal in that school. In this school they have used the ORID reflective process in all grade level meetings and have instituted gallery walks, where teachers open up their classrooms to the rest of the faculty. The ORID structure has been less successful during the school year than during the summer; as teachers admit, during the summer they were required to meet with their classroom teams and submit an ORID sheet at the end of each day. These structures were not in place during the school year, and therefore teachers were not willing to add them to their existing schedule.36 Improved Communications • Many teachers have integrated the structured parent observations into their school's parent day, making parents more aware of what makes a successful classroom. On Visitors day, parents have a conversation with the teacher and administrator about the format of the visit and about what they should be looking for as they observe the class. In addition, parents are handed an observation checklist that they take with them to their child's classroom. During their visit, teachers are escorted by student teacher guides. After the observations, the parents and student teacher guides break into small language appropriate groups to discuss their observations. 35 36 Interviews conducted with participating teachers, summers 1999 and 2000. Interviews conducted with participating teachers, summers 1999 and 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 51 After the small group discussions, the small groups report out to the whole group. This has been a powerful experience during the summer clinical model and has been successfully integrated into the school's parent day. See e-mail for rewrite of this paragraph. Classroom Practice and Technology Literacy • Teachers reported that the summer clinical helped them improve their standards-based assessment, which in turn had a significant impact on student learning during the four weeks of the summer clinical model program. The summer program modeled performance-based assessments for students, which all teachers thought helped them see where their students were in the beginning of the session and helped them assess the gains that students made by the end of the summer. One teacher reported, "I thought the written assessments were great. It helped teachers learn how to do authentic assessment. I had only one kid pass in the beginning and at the end every kid could do it. In the beginning, I asked them to tell me everything they could about measurement and at first they could not do it and by the end they could." This teacher thought that the workshops that were provided during clinical model taught teachers about writing assessments and how to practically integrate standards into their assessment. This teacher reported intense and often difficult conversations about authentic assessment and thought that it was an important experience for the whole group. She felt that it provided a space for teachers to have conversations about assessment and student learning with colleagues as well as university faculty.37 The Portfolio Tool The portfolio tool was piloted during the summer 1999 clinical summer program and was implemented in summer of 2000. The tool required teachers to complete an introduction describing their school and classroom, a lesson plan tied to district standards that they implemented with their student in the summer clinical, and an action plan for the upcoming school year. However, only the elementary school teachers created complete portfolios while middle school teachers were provided with a scaled down version of the portfolio. Due to minor technical difficulties some teachers completed portfolios using the portfolio tool, while others completed their portfolios using a word processing program. Based on criteria established by the project team, CCT scored teacher portfolios. The content analysis reviewed and scored portfolios to determine whether teachers had included all components as outlined in the portfolio tool. The project team received a total of 40 portfolios, (out of 49). The project team rated each portfolio as (1) excellent, (2) average and (3) poor. Of the 40 portfolios, 22 of these portfolios were completed using the tool, the others were produced using a word processor: 37 Interviews conducted with participating teachers, summers 1999 and 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 52 • Nine were excellent (6 of the 9 were created with the tool); • Twenty were good but did not include some pieces of information required by the tool; and • Eleven were poor. None of the portfolios that were rated poor were created using the tool. Professional Development • The portfolio tool provided the structure for a reflective process that teachers did not engage in if writing their own portfolio.38 Some teachers did not complete portfolios using the tool because of technical difficulties. Although some teachers completed portfolios using word-processing software, many used the structure that was provided by the tool. Specifically, the tool requires teachers to: provide an introduction that presents the schools culture and context, connect the goals and activities of their lesson plans with district standards, and focus on professional standards. The tool facilitates this connection by providing a menu of standards that teachers can automatically attach. The tool also requires teachers to reflect on the lesson plans, create action plans that directly address student needs, and identify areas where “they can strengthen their own practice.” The portfolios that were simply typed were not as comprehensive. Teachers did not articulate their curriculum and learning goals clearly or identify areas in their teaching practice that they would like to improve. As the tool is implemented in future summer clinical models, as well as in other professional development programs, it will help support a more reflective process for teachers. It will also provide a structure for teachers to outline their teaching and learning goals clearly and identify an action plan to achieve those goals.39 Effective Administration • In addition to the integration of the portfolio tool in the summer clinical model schools, the tool was expanded to support the BTSA program. This program provides mentors to first and second year teachers. The BTSA teams are required to complete state forms as they progress through stages of the program. These forms were put online and added to the portfolio tool. The development team introduced the tool in December 1999 to ten BTSA teams. All teachers thought it was an enormous benefit to be able to fill out the required paperwork online because it made the process more efficient. Additionally teachers are able to access this tool from home. This is an important benefit because teachers, especially beginning teachers, do not have time during the school day to write and reflect. The only criticism was that some of the beginning teachers had already completed the second and third levels in the BTSA process and those forms (CFASST 2.1-2.6) were not yet available at that time. 38 an initial analysis of portfolios that were completed using the tool and those that did not. 39 Content analysis of teacher portfolios, Fall 2000. Researchers received all portfolios that were completed during the summer clinical model and conducted IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 53 Teachers reported that they communicated with their coaches via email and in face- to-face meetings. All of the BTSA teachers responded that having their coaches review their CFASST forms online was an important benefit because of decreasing in-person time with their coaches due to conflicting schedules. The tool helped facilitate on-going communication. They were also able to receive their coaches' feedback in a timelier manner.40 Evidence of Institutionalization • San Jose has fully integrated the summer clinical model into their existing professional development initiatives. The district is now integrating the portfolio tool and the summer clinical model into all district programs for pre-service and new teachers. The training and support of pre-service teachers is a priority in the district because of the high percentages of new teachers that will be placed in San Jose schools over the next several years. In addition, the project coordinator, who has coordinated the Reinventing Education project since its inception, has increasingly taken on more responsibilities at the district. She is now coordinating all programs for pre-service teachers. Therefore, she is in a central position to integrate the clinical model and the Learning Village Tools. The district has also identified an additional staff person to coordinate the integration of the portfolio tool into BTSA. • The superintendent considers CLASS as part of an overall professional development strategy that focuses on experimentation, reflection and collaboration and stated that the clinical model will now be part of all SJUSD professional development programs. CLASS supports BTSA and the Professional Development Schools. She commented, "I am very pleased with what this model has provided for the district. What we have achieved is the affirmation that professional development needs to be teachers working together in a clinical model. They can reflect, create and implement curriculum together." She further stated that the clinical model would now be part of all SJUSD professional development programs.41 • The district has increased support for the summer school model with a 21st Century State BayTech grant. The Summer Clinical model has been a successful model that has been fully integrated into the SJUSD professional development program. Over the three years of the project, the summer clinical model has grown from one summer elementary school to four schools - two elementary schools and two middle schools. The new grant will extend this model to additional schools and after school programs. • The Director of Professional Development is setting aside district resources to purchase more licenses for the Learning Village. The district wants to expand the use of the tool to help support the e-mentoring program that is being instituted this year. Students from throughout the district will be paired with an adult professional 40 41 Summary feedback forms distributed to BTSA teams, spring 2000. Interviews conducted with the Superintendent and Curriculum Coordinator, winter 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 54 from the community. The district plans to integrate the communication tools available through the Learning Village to support this initiative. Ongoing Challenges The Summer Clinical Model Although the summer clinical model has proven to be a successful professional development program for the San Jose district, it faces some challenges. • The summer clinical model requires strong commitment from local universities. Although two new universities participated this year, the program coordinator had some difficulty recruiting university students from San Jose State. In addition, she had to act as an advisor for San Jose State students because the faculty member who had committed to this responsibility left the university prior to the summer, and the university had not reassigned it to another faculty member. • The Project team requires cooperation from all summer clinical model schools. The summer schools were successfully implemented in the two elementary schools using the model’s design. Classroom teachers collaboratively designed curriculum throughout the school year and were able to implement it in the classroom with their participant and student teachers. However, the middle school teachers, while they were assigned student teachers, did not have a one-to-one ratio between student teachers and participant teachers so some were unable to fully experience team teaching in the classroom. Prior to the summer school, the district notified the project team that the middle school was going to target students who were “at-risk” of retention and were given prescribed curriculum that they had to implement. Teachers were not able to collaborate or design curriculum as they had expected. The project team also did not receive full support from the administration of one of the middle schools.42 The Portfolio Tool • The district needs to help teachers understand the importance of the portfolio process in improving their instruction. Although the portfolio tool was more fully integrated into the summer program, there were still teachers who were unclear about its objectives and preferred their own methods of documenting their teaching practice. Several teachers faced technical difficulties when trying to use the tool, which caused some frustration. In addition, there were teachers who had little or no experience documenting and/or reflecting about their teaching practice in writing. Many teachers reported that they did not see a need for portfolios unless “they were looking for a job.” 43 42 43 Site visits conducted, summer 2000. Interviews conducted with participating teachers, summer 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 55 San Francisco Unified School District District Background The San Francisco Unified School District is the fifth largest school district in California. 2000-01 Data ü Number of Students: 62,569 û Seventy percent of students are minority. ü Number of Schools: 115 û Elementary 77 û Middle 17 û High 21 ü Number of Classroom Teachers: 2,671 Budget (1999-2000): $432,931,000 Solution Summary The Reinventing Education project in San Francisco has developed Student Success Team Case Management Tools to support the reform of special education in the San Francisco Unified School District (SFUSD). Specifically, the San Francisco Reinventing Education project team, in collaboration with the Every Child Can Learn Foundation and IBM, created applications that support the existing Student Success Teams (SST) process. The Student Success Team is a school-based team process first instituted in 1985 by the California Department of Education. SSTs identify students in need of extra resources in the classroom or requiring referrals to special education. The SST case management tools are a suite of applications that includes the following: • Referral Tool – A Web-based tool that teachers use to refer students to the SST process. These forms are designed to automatically pull student information from the district server. The referral includes students’ strengths, the teacher’s concerns and strategies that have already been attempted by either the teacher, counselor or caregiver. • Meeting Wizard – A laptop-based tool that a SST team member uses to record the SST meeting. It is described as a workflow model that assists participants in documenting necessary information. The data fields in this tool correspond to the state guidelines for an effective SST meeting. The data collected in the Meeting Wizard is uploaded to the server, providing a file for each child that includes the referral and documentation from each SST meeting. • Strategies Database (still under development) – This database will include academic, behavioral, and health strategies. Teachers and SST team members can use the database as a resource both before and during the SST process. Additionally, these IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 56  tools will give SST team members the ability to track effective SST strategies. The Strategies Database will also be used for teacher professional development. Although SST meetings are conducted differently at each school, SST meetings always bring together a group of individuals to focus on the student. The state guidelines identify the SST team to include the following: a parent, the student’s classroom teacher, the resource teacher, and the principal. SST is a very time-intensive process and requires extensive documentation. Often, schools are unable to retrieve necessary data about a student who is being referred to the SST process. The SST tools are designed to help facilitate the meeting by providing the meeting coordinator with student information during the meeting. In addition, the SST case management tools help to ensure that the SST meetings focus on the strengths of the students, include strategies that have been proven effective, and provide every member on the team with information that is more comprehensive than was previously available to them. Focus of Evaluation In collaboration with the San Francisco project team, CCT identified the following indicators to track through the three years of the project: • General use of the SST tools; • Understanding and expectations of the project and tools; • Attitudes toward the tools; • Use and effectiveness of the tools; • The role of the SST coordinator in the implementation of the SST tools; and • Institutionalization of the SST tools within the district. Current Status The current design of the SST tools resulted from a close collaboration between SST coordinators from the two initial pilot sites, and the Reinventing Education project team during the development phase of the project. Primarily, these SST coordinators were very involved with the development of the Meeting Wizard. In the beginning of the pilot year only one school was using the Meeting Wizard to record SST meetings, the other used the tool, but only to enter their notes after the meeting. By the end of the school year, the Meeting Wizard had been redesigned and both coordinators were using it during the SST meetings. It was redesigned to work in offline or stand-alone mode in its second iteration, therefore speeding up its performance considerably and making it more reliable as it was no longer necessary to rely on a connection to the district server. One SST coordinator commented, “I think the whole meeting has improved. There is more adherence to the agenda; it has a certain amount of weight, with the computer in the room. Everyone who participates in the meeting has a lot more self-awareness. The whole process has become more official. It really has set the agenda, now we tend to get through the entire agenda. I have as much room as I need, they’re more complete, also to assign responsibilities and IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 57 due dates, there is much more follow through from the SST participants.”44 He added that the IBM design team was very responsive to his concerns and suggestions. Currently 550 educators use the SST tool at 19 schools (22 schools by the end of the school year). This number does not include the staff at several new schools that are now being entered into the system. The grant site project manager continues to demonstrate the tool to new schools in order to recruit additional schools. The first roll-out phase of the project began in spring 1999. The project team prepared a roll out plan to spread the use of the tools to an additional 10-15 elementary schools. The project team delivered several presentations to the district and elementary school principals in order to select schools for the scale-up phase of the project. Eleven new schools were selected to be trained during three sessions. The elementary schools that were selected were already in the process of reforming their SST process (or were already successfully conducting effective SSTs), and had a technology infrastructure that could support the use of the new SST tools. Participating schools were required to send teams that included the principals to these training sessions. These teams were then responsible (with support from the project team) to train their teaching staff to use these tools in the 1999-2000 school year. The 13 pilot schools integrated the tools into their SSTs. However, there were technical difficulties that prevented full implementation. Primarily, the tools are designed to pull data from the district’s student information system, however this component of the design was not functional until summer 2000. The district installed a new student information system in January 2000. The project coordinator reports that the technology is now working well and pulling data from the district’s student information system. He is closely working with the existing 11 roll-out schools to make sure that they are using the tools. Specifically, he is retraining SST teams that had not used the tools due to technical difficulties and conducting additional training in schools that have new SST teams in place. The project team encountered technical difficulties throughout the development and implementation phase of the project. This was partly due to on-going improvements in the software suggested by the San Francisco project team, and the difficulty of accessing the district server. Both participating schools had significant problems connecting to the district server and were often disconnected. The referral tool and the Meeting Wizard interact with the district server to download existing student data and upload the referrals and meeting notes when they are completed. This problem continued throughout the pilot and the scale-up phase of the project. Obtaining effective cooperation with the district's IT department remains a top priority if the tools are to work as planned. The MIS department, and IBM designers worked very closely throughout the spring to ensure that the SST tools communicated with the new student information systems put in place in the district in January 2000. In the last year there has been growing district and state interest in the integration of the SST tools into all schools. The MIS department, 44 Interview with SST Coordinators from Pilot Schools in spring 1999. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 58 initially unable to give full attention to the project, is actively working with the project team to iron out any further technical problems.45 The project coordinator is closely working with the roll-out schools to make sure that they are using the tools. Specifically, he is re-training the SST teams that had not used the tools due to technical difficulties and is conducting additional training in schools that have new SST teams in place. Currently, the project team is not sure what district resources will be used to support the implementation of this SST tools throughout the district. However, given the central role the SST process will play in the new strategic plan at the district, the executive director of the Every Child Can Learn Foundation, a key partner in the Reinventing Education project, is confident that the district will support the technical maintenance and training necessary for the tools to be successfully utilized in the district. Presently, all district departments and personnel are in the process of restructuring so there is uncertainty throughout the district. Evidence of Benefits Effective Administration • The SST tool training and implementation helped support an overall reform of how the SST process was being conducted in participating schools.46 Because the training focused on the importance of SSTs as offering alternative strategies for students and teachers other than immediate referral to special education, the majority of teachers and SST coordinators reported changed perceptions of the SST process in their schools, now seeing the process as providing students the services they need.47 This is an important result particularly because the district had drastically reduced funding for the SST process in most schools. The majority of schools in San Francisco did not have a full-time SST coordinator. This policy has been revisited and there is increased interest and plans to conduct district-wide training on effective SSTs. In the spring of 2000 a survey was distributed to all teachers in the participating schools. Of the 197 teachers that returned surveys, the majority 63% (124 teachers) reported that “the new electronic system for referring students had enriched the SST process.”48 Effective Administration • The SST tools are supporting an overall reform agenda that has focused on making the SST process effective for all students. The development of the SST tools was very responsive to the needs of SST coordinators and teams. As more SST teams use the tools, the documentation will be available district-wide, which will 45 46 Finding is based on interviews with SST coordinators and surveys of teachers in the 11 participating schools. 47 48 Interviews with all participating SST coordinators in spring 2000. Surveys distributed to all participating teachers in spring 2000. Information gathered from ongoing frequent communication with Phillip Bryan spring-fall 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 59 provide information electronically and allow for school- and district-level analysis of the SST process. The tools allow teachers and SST teams to send referrals and record SST meetings on-line and ensure that this information will always be accessible. It will also provide the district with data that they need to make informed decisions about students’ needs. In light of the history of the SST process in the district, everyone involved in the reform is committed to providing parents with access to information. • Although most schools have only partially used the electronic tools to document their SST cases, both SST coordinators and teachers reported that the SST meetings are more effective than before the use of the tools. The majority of teachers (62%, 120 teachers) reported that “this year the SST process has provided more support and classroom strategies for my students that in previous years.”49 The SST coordinators are very enthusiastic about the Meeting Wizard and its ability to project the meeting notes for the whole team. Several SST coordinators described parents as “empowered” because they can see the information that is recorded about their child. As one coordinator remarked, “Using the technology makes the information accessible to the whole team and the family, they feel empowered. It helps us keep the information organized. We have a paper trail but for the most part we can access the system. The technical problems we had were district problems but lately things went smoothly.”50 • Teachers believed that the electronic tools had a positive effect on the SST process. In a survey distributed to teachers before the implementation of the SST tools, teachers were asked about their expectations for the electronic system. Most teachers thought that the new SST tools would reduce the time between referral and the meeting. For example, one teacher wrote, “In the past it has taken up to six months before an SST has materialized. It seems the new system will expedite the process.”51 Although some teachers were apprehensive about the new system because of their lack of technological expertise much of their apprehension dissipated. In a survey distributed at the end of the year, 60% of teachers (118) reported that, “the new electronic system has enriched the SST process by making it more useful for classroom teachers and addressing a broader range of student needs.”52 • In light of the stronger SST process brought on by the integration of the tools and the focus on improving the process overall, teachers in the schools are now actively utilizing the SST process. This is seen as an indicator that the reform of the SST process is having impact on teachers’ perceptions of the process. One coordinator said, “the number of referrals have been growing, before only a few teachers would refer students, now teachers are utilizing the process. Part of the problem was that there was nothing happening as a result of the SST. Now the district is supporting our process. Now the teachers know they have to really try the 49 50 51 52 Surveys distributed to all participating teachers in spring 2000. Interviews with all participating SST coordinators in spring 2000. Surveys distributed to all participating teachers in fall 1999. Surveys distributed to all participating teachers in fall 1999. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 60 interventions. The district is listening to the teachers when they exhausted all the strategies.”53 On average, schools reported that between 30 and 50 referrals are initiated each year. In one school there were 72 active cases, but many of these students were carried over from the previous year. • Also as a result of the strengthened SST process, teachers were positive about SSTs within their schools. They felt that the process was an effective strategy for their students. The majority of teachers (70%, 134 teachers) disagreed with the statement, “The SST is only for students that need to be in special education.” In addition most teachers (80%, 158 teachers) disagreed with the statement, “I still don’t receive any help or strategies as a result of the SST process.”54 These responses indicate that teachers’ perceptions have changed through the course of this project. Specifically, in surveys distributed in the beginning of the year, more teachers reported that they did not receive much support from the SST process and felt that the SST was for students who should be referred to special education. When asked to describe a successful SST process, teachers often wrote, “finding the appropriate resources for a troubled student.” For example, one teacher wrote, “Last year, the SST for my student succeeded in assigning a social worker to oversee the home situation and helped his mother with strategies. (I also met regularly with the social worker and gave suggestions.)”55 Improved Communications • The tools help scaffold the SST process for schools that are still reforming their SST process. One principal reported, “The training helped my staff see the importance of an effective SST in supporting them as well as their students in the classroom, and as an effective way to communicate with parents.”56 Both coordinators and teachers were enthusiastic about having information housed on a central database, thereby eliminating the paper chase they had previously experienced. The tools now interface with the district’s student information system so teachers and school SST teams will have instant access to student information, which is especially important if the student is new to the school. Evidence of Institutionalization • The current project team has made the full implementation of the SST tools in the district their primary goal. The current project team, which includes the director of the Every Child Can Learn Foundation (ECCLF), a program officer from ECCLF, the program director for Elementary Special Education Services, and a school psychologist from Special Education Services, has made the full implementation of the SST tools in the district their primary goal. This team meets regularly to discuss 53 54 55 56 Interviews with all participating SST coordinators in spring 2000. Surveys distributed to all participating teachers in spring 2000. Surveys distributed to all participating teachers in spring 2000. Interview with principals with participating schools in fall 1999. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 61 implementation strategies as well as issues around technical infrastructure. During the 1999-2000 school year, the project team worked hard to involve all departments throughout the district in the roll-out phase. This strategy has shifted this year, primarily because the administration at the district has changed. A superintendent, who formerly worked in Washington D.C and Seattle, has become the new superintendent in San Francisco. She has begun an ambitious restructuring program that is effecting all departments in the district. The superintendent’s primary goal is to restructure the top levels of the district’s organization and to develop a strategic plan accompanied by an Information Technologies (IT) plan. According to the executive director of ECCLF, who is also a member of the superintendent’s strategic planning group, the SST process will be an element for the strategic plan to which all schools will be held accountable. Therefore, principals in every school will have to implement an effective SST process that will disseminate classroom strategies to meet the needs of all students. The IT plan that will be developed this fall will support this new district emphasis on the SST process and the SST tools will play a central role in this plan. 57 Ongoing Challenges • The district needs to provide ongoing training to help teachers better understand the importance of the SST process. There are teachers that continue to be frustrated with the SST process. For example, when asked to describe a negative experience, some teachers wrote that the SST process in their school was unable to deploy resources for their student in a timely manner. One teacher reported, “most times nothing really resulted from those lengthy meetings that I felt was helpful to the child or for me to help the child.” Other types of negative experiences with the SST process resulted from parents not implementing the strategies that had been suggested, “My student’s SST has not resulted in his dad getting him to school on time.”58 • The Strategies Database is still under development. Last year the project team was working with the Department of Curriculum and Instruction to populate the database with academic strategies. Due to restructuring, this has been put on hold. Presently there is no clear plan to populate the strategies database. Although the strategies data base is not essential to the functioning of the SST electronic tool, it was intended as an additional resource for teachers and administrators to provide support and appropriate interventions for students and provide a means to share effective practices with teachers. • The district needs to put in place plans for district-wide implementation of the tools. As mentioned above, the new superintendent is in the process of restructuring all departments. When these changes are in place, the district plans to roll-out the tools to the remaining elementary, middle and high schools. 57 58 Interview with Dana Serleth, October 2000. Surveys distributed to all participating teachers in spring 2000. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 62 Vermont Department of Education State Background ü Total School Enrollment (1999-2000): 104,559 ü Number of Schools: 335 û Elementary (K-6): 167 û Elementary (K-8): 64 û Middle (5-8): 24 û K-12: 13 û Middle/High (6-12): 28 û High: 25 û Other: 14 ü Number of Teachers: 6,684 Total Expenditures: Solution Summary $858,401,862 The initial goal of the IBM Reinventing Education initiative in Vermont was to develop technology tools to support and augment Standards Into Action, the state's standards initiative. The original plan called for the development of three software tools: a unit of study tool for the creation of standards-based curriculum units, a scoring tool for rubric- based assessment of student work, and a sharing tool to facilitate communication and collaboration among teachers. Ultimately, the technology solution was comprised of two online tools: • Authentic Assessment Tool, which evolved from the scoring tool, assists teachers in developing skills in using rubrics for scoring student work, and in applying those skills to the work of their own students • Instructional Planner, the tool originally developed as part of Reinventing Education in Charlotte-Mecklenburg, steps teachers through the creation of instructional units, with the capability to search and reference relevant standards, attach resources, and link to related Web sites. Teachers can choose colleagues to act as online reviewers, and can upload their units to a shared database. IBM discontinued work on the unit of study tool when the Vermont project team agreed that Instructional Planner, with some modifications, would be a suitable alternative. Over the course of the project, the functions of the sharing tool became integrated into the other software components, making a separate sharing tool unnecessary. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 63 During the early phases of the project, RE supported the continuing effort to introduce teachers to the standards and the development of a set of print materials, the Standards Into Action Professional Development Toolkit, for this purpose. This work coincided with early software development efforts. The software was to be distributed on CD-ROM for both Mac and PC platforms. Midway through the project, it became clear that the Web was becoming a powerful environment for accessing and sharing electronic tools and information, and the project team decided to port the RE tools over to the Web-based Wired for Learning environment, which the Vermont team refers to as Standards Into Action. Development of the Web-based version of the Authentic Assessment Tool was delayed, and as a result the focus of much of the project's outreach was on introducing educators across the state to the Instructional Planner. Every school in the state was invited to identify a teacher or specialist as a Standards Into Action liaison. Liaisons participated in training sessions conducted by the State Education department, and in turn, trained teachers in their home districts and regions to use Instructional Planner. Initial Standards Into Action outreach included introductory and hands-on training sessions for 500 liaisons. Focus of Evaluation At the outset of the evaluation, the Center for Children and Technology worked with project leadership to develop a set of indicators against which to measure the progress of the initiative. The researchers then developed a plan for collecting data related to the indicators. The indicators we examined in Vermont were: • The degree to which teachers are aware of the Vermont standards; • The degree to which teachers understand the goals of the project; • The number and types of professional development opportunities and print materials available to educators; • The extent of teachers' knowledge and use of the tools, and teachers' attitudes toward the tools; • How effectively the tools meet educators' needs; • The degree to which schools' culture changes as a result of educators' participation in the project; • The degree to which the state assumes ownership of the initiative; and • How effectively the state implements scale-up of the initiative. Current Status59 Liaisons are conducting follow-up Instructional Planner trainings for teachers already familiar with the application as well as introductory sessions for new teachers. In 59 phone interview with Doug Snow, December 7, 2000. Per site visit meeting with Doug Snow, Doug Walker, and Marian Lawler, October 4 and 5, 2000, and IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 64 addition, the state has established a pilot program to conduct intensive professional development and technology integration activities in participating schools. Fifteen schools have been identified to participate in the pilot program. Schools were invited to apply by filling out an application form, and selections were made based on several factors: • the school's willingness to involve at least 75% of its staff in the project; • the school's commitment to produce at least one instructional unit to be shared with project staff, along with various other materials and resources; • the degree of connectivity (minimum 56K fulltime access and preferably a T1 line); • the level of teacher access (a lab that could accommodate at least one quarter of the school staff at one time and/or a computer in every teacher's classroom); • the number of other initiatives the school was engaged in (some applicants had as many as seven other professional development initiatives under way, which the pilot program staff felt was too many to allow for the level of commitment they wanted); and • geographical distribution across the state's five regions. The schools range from novice to expert in the areas of both use of technology and application of the standards. Each school undergoes a needs assessment regarding teachers' technology use and the familiarity with the standards, and the results are used to identify appropriate starting points for professional development. In addition to the in-depth work in the pilot, the Department of Education's State Assessment Team is working rolling out the Authentic Assessment Tool across the state to all schools with a focus on local assessment efforts. In addition, a number of teaching colleges have expressed interest in using Learning Village in their pre-service programs, and several have attended recent Learning Village workshops. Additionally, the system is being adjusted to accommodate all potential users in the state. In order to use the authentic assessment tool effectively, teachers have to register all the students in their classes. While system capacity was expanded enormously to handle the initial roll-out of the tools in Vermont, it still could not accommodate all the teachers and students in the state. IBM staff have expanded capacity still further in a version of the software that is currently being tested in Memphis, and which is expected to be implemented in Vermont in mid to late January 2001. Evidence of Benefits60 Professional Development • 60 Reinventing Education tools facilitate teacher collaboration on instructional units. Liaisons who have held trainings in the last year report that school staffs have Per site visits and interviews with State Department of Education and IBM staff, liaisons, and teachers. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 65 increased professional communication and joint planning through use of the tool in these sessions, even if tool use has not yet become a permanent part of their planning. One technology coordinator reported, "It’s useful to have a resource that you can access any place any time — teachers in different schools are collaborating on units." In effect, the process of working together in the trainings has increased communication among teachers. Improved Communications • Over six hundred teachers have created home pages using the tool as an explicit means of providing parents and students with classroom information. The rapid spread in the use of Home Page Creator, to such an extent that it required a redesign of database structure holding these home pages in Vermont, indicates that this has been an immediate way into the tool suite for many teachers, though pages are not necessarily updated regularly. Even schools that have their own Web sites for internal or public access are using the tool and then linking to the pages created through it. The Web pages provide parents and students with information about what is going on in class, homework assignments, and upcoming events such as field trips. Classroom Practice and Technology Literacy • The Standards Into Action tools facilitate access to, and use of the standards. While there were systematic distribution efforts and workshop offerings to familiarize all teachers with the Vermont Framework of Standards and Learning Opportunities, the workshops on using the tool are the first time many teachers begin actively relating the standards to what they do in their classrooms. For those participants, awareness of the standards alone had not had an impact on their teaching. In stepping them through the process of developing a unit plan with direct connections to the standards, the STANDARDS INTO ACTION workshops offered them a vehicle for applying the standards to their daily classroom practice. Evidence of Institutionalization61 • There are 6,700 accounts on the system — over half of the state's K-12 teachers. While only a subset of these are active users of the Instructional Planner tool or other features such as Home Page Creator, Vermont has developed a school liaison infrastructure in most schools along with assigning a teacher training and support person. These liaisons in some case are taking active responsibility to introduce and train their fellow teachers on the use of the Standards Into Action tools. • The creation of a standards-based unit of study will be required as a part of the teacher re-certification process. The Standards Into Action program retains broad support and is being tied to key accountability actions in the state. The program director retains active involvement with the project even with new responsibilities as 61 Per site visits and interviews with State Department of Education and IBM staff, liaisons, and teachers. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 66 assistant commissioner in the state where he is using the program to target specific school improvement initiatives with at risk schools and well as the general scale up efforts statewide. • Standards Into Action is a central component of new grants and initiatives. It is a core strategy within the PT3 program launched in the fall of 2000 by the University of Vermont College of Education and Social Services. The goals of the grant are for education faculty to prepare pre-service teachers for standards-based instruction, and for pre-service teachers to use technology to create and maintain electronic portfolios documenting their development as educators. Standards Into Action is a major focus of the project, and professional development activities include training for UVM faculty on using Learning Village62 tools. Faculty members will use the tools in assessing their students' work, and will teach pre-service teachers how to use them as well. • Teacher education institutions are becoming involved in the initiative. In addition to the PT3 program, several teacher education colleges, including St. Joseph's College, Johnson State College, and Lyndon State College, are using Standards Into Action. Since a majority of Vermont trained teachers stay in Vermont to do their teaching, this insures that many of the new teachers entering Vermont schools will be already familiar with the tools. Ongoing Challenges63 • The user interface for both the Instructional Planner and for the Authentic Assessment Tool is perceived by many of the users as unnecessarily cumbersome. As an example, an art teacher who is the STANDARDS INTO ACTION liaison at Spaulding High School in Barre, VT, conducted a week-long Instructional Planner training for teachers at her school in the spring of 2000. At the end of the week, she asked teachers to write down their reflections, which suggested that "the site could have a lot of potential, but right now it’s too difficult to use in terms of how you maneuver, how you organize. For instance, standards appear in your unit in the order you go get them, and there’s no way to reorganize them." • Some teachers are reluctant to submit the units of study they create to a state- wide pool of lessons. Liaisons report that some teachers are initially resistant to sharing their lessons via the STANDARDS INTO ACTION database. This was just a few teachers and while they did state that these teachers are not prepared to share their early efforts, the liaisons report that these teachers say they would be more likely to share later work once they are comfortable with the tool. • The tools require intensive Web access. Many teachers do not have consistent access to Internet connections in their classrooms, where they are most likely to do 62 63 Vermont project leaders use “Learning Village” and “Wired for Learning” interchangeably. Per site visits and interviews with State Department of Education and IBM staff, liaisons, and teachers. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 67 planning. Training sessions involving large groups of teachers have been problematic because of limited bandwidth connections in the labs or rooms where the training or access takes place. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 68 West Virginia Department of Education State Background ü Public School Enrollment (Second month, 1999-2000): 285,785 ü Number of Public Schools (September 2000): 818 û Elementary 607 û Secondary 211 ü Number of Teachers: 20,988 Budget: $2,177,834,065 Solution Summary The West Virginia Reinventing Education project was designed to address concrete, identifiable challenges within the state educational system by utilizing the knowledge and experience of classroom teachers as it engaged them in the development of curricular materials. The materials development process used in the Reinventing Education program required that these materials integrate the use of technology, specifically the Internet, address state standards, provide relevant assessments and take into account the degree to which access to technology varies across classrooms. At its most basic level, the goal of this program was to create a body of technology-rich classroom materials that could be used by teachers in diverse classrooms across the state to address deficiencies in student learning as identified through state testing. On a deeper level, however, the program recognized that to meet project goals Reinventing teachers would be constructing materials that integrate pedagogy, practices and tools of inquiry that many had not previously considered or experienced. Focus of Evaluation Through the Reinventing Education evaluation, carried out by the Center for Children and Technology, a number of features were identified for use as indicators of success for the program as it evolved. Specifically, these were: • Changes in the teaching practices of Reinventing teachers with regard to classroom use of technology; • The extent to which teachers both developed and implemented Reinventing materials in their classrooms; • The extent to which the project was able to facilitate professional development through an increase in online professional communication, use of Web-based resources and the lesson jurying process; IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 69  • The degree to which the first cohort of Reinventing teachers facilitated collaboration among other Reinventing participants; • How effectively Reinventing materials address the state Instructional Goals and Objectives (IGO's), which was to be analyzed through changes in student test score data; and • How seamlessly the program was able to scale-up to accommodate new participants and new disciplines. Current Status The West Virginia Reinventing Education program has brought three cohorts of teachers through the materials development and implementation process that was designed as part of the site's solution. The program began in its first year with a pilot group of 12 math teachers, and in the second year extended out to recruit a larger group of 34 teachers within this same discipline. The final phase of the solution implementation involved a roll-out to a third group of participants that represented 35 teachers from three new disciplines: science, social studies and language arts. Teachers in each phase of the program were required to carry out the same kinds of activities, including developing lesson plans around Internet resources, field testing lessons, and submitting lessons to a jury process for review and revision. The experience of each group differed, however, in relation to the stage of development of various components of the program, and the program has grown and refined itself with each new group. Currently, West Virginia's program manager has a committed sub-group of teachers who have an understanding of the pedagogical goals of the program and the processes in place to help other teachers meet those goals. As such, these teachers are being utilized to help implement other grants that are using the Reinventing solution as a model for professional development within various levels of the education system. This outreach program, call PD I, has extended the Reinventing work to an additional 173 teachers statewide. In turn, 147 of these trained teachers returned to their own schools where they have conducted local training sessions for a combined total of 1470 classroom teachers. Evidence of Benefits64 Impact on Student Learning Our analysis of results from state standardized tests indicated that use of lesson units created through the Reinventing Education project helps students who are under-performing make substantial gains in a relatively short period of time (within a single school year). For students already performing above the average, use of these lesson units maintains their performance lead. This study demonstrates these results across grade levels and across curriculum areas. 64 administered during 1998-99 and 1999-2000 school years. All references to comments and findings are based on interviews conducted in spring 2000 and surveys IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 70 The units and lessons developed during the first three stages of teacher lesson plan development are directly tied to specific West Virginia Instructional Goals and Objectives (IGOs) related to areas in the core curriculum subjects. Analysis of annual standardized achievement testing has been used to decide what specific skills and concepts should be focused on in these lessons as indicated by where West Virginia students are having difficulties. Since these lessons are highly targeted and the state testing program is well-aligned with the IGOs, we were able to look at the impact on student performance as measured on the Stanford Achievement Test (SAT-9) when teachers made extensive use of one of the units that had been developed, juried, revised and placed in the best practices section of the Instructional Planner. Teachers who had participated in the Reinventing Program in one of the three initial cohorts where surveyed for level of use of lesson plans from the Instructional Planner database during the 1999-2000 school year. Those teachers who used lesson plans from this the Instructional Planner actively with their students were selected for this current impact study. Earlier research by CCT during the 1997-98 school year with the mathematics teachers in the initial pilot development group indicated that participation in the unit and lesson development work did not necessarily lead to student performance improvement on standardized achievement tests unless there was also active use of the unit with students. Teachers who used the units did see a statistically significant performance gain in achievement tests scores on the relevant sub-tests in comparison to a matched control group. In other words, this earlier study indicated that achievement gains were attributable to use of the lesson plans and not some effect created as an artifact of the teachers of these students being singled out for specialized treatment through their participation in the Reinventing program. The earlier study indicated that while participation led to professional growth for teachers and better understanding of the state’s IGOs and of innovative lesson development, and while this experience should lead to instructional benefits in the long run, there was not short-term impact on student learning unless the units were actually used with students. In this new follow-up study, we looked at whether students who worked with units during the 1999-2000 school year would show significant improvements in their Stanford Achievement Test scores at the end of the year. In this study, West Virginia teachers now had developed and juried lesson units in Language Arts, Social Studies, and Science in addition to Mathematics and we were able to look at impact in all of these areas. After identifying classes in which the lessons were used, we obtained standardized scale scores for sub-test areas from the spring 2000 state-wide administration of the SAT 9 for each student. Their standardized scale score results from the previous year’s SAT 9 were also collected and used to look at the gains achieved by these students. This provided a treatment population of 438 students in grades 7 through 11. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 71 Sixty students at each grade level 6 through 10 were selected at random from counties that matched the state mean on the spring 1999 SAT 9 to serve as a control group. These students’ standardized test scores were also collected for the spring 2000. If there was no subsequent test score available because the student moved or did not take the test, that student was dropped from the control group. A series of Repeated Measures Analyses of Variance were used to examine differences in the 1999 scores and the 2000 test scores for the students who used the lesson units developed using the Instructional Planner as compared to the control group of students. A Repeated Measures Analysis technique was used rather than a simple comparison of gain scores (i.e., t-tests) because of concerns about violating the assumption of normality which underlie the use of t-tests. The Repeated Measures analysis provides information about both main effects and interactions. In the present analysis there was one main effect that represented the overall differences between the students who attended classes where the lesson plans were used and students in the control group on the combined 1999 and 2000 Stanford Achievement Scores. There also was a main effect representing the difference between the 1999 and 2000 achievement test scores across both groups combined. The interaction effect, the information most relevant for examining the effects of the use of lesson plans from the Instructional Planner, represents the differences between the pre and post test scores for the Instructional Planner (IP) and the control groups separately. Table 1 includes the average standard score gains for students who used lesson units in the Instructional Planner and for the control group. The F-value presented in Table 1 refers to the F-value calculated for the interaction effect for each of the Repeated Measures Analyses. F values for the main effects are included in the following description of the evaluation results. Standard Score Gains in Reading, Language, Spelling, Mathematics, Social Studies, and Science for students using lesson units in the Instructional Planner and the Control Group ** = significance at the .01 level * = significance at the .05 level Grade Test Score Used IP Mean + SD Control Mean + SD N of IP N of Control F 7 Reading Total 18.05 (18.87) -4.62 (45.06) 43 39 9.12** Language Total 6.05 (22.20) -13.60 (58.59) “ “ 4.47* Study Skills Total 13.56 (29.03) 10.53 (56.16) “ “ 6.15* IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 72 8 Reading Total 15.28 (19.62) 14.82 (33.58) 36 60 .01 Language Total 14.50 (20.75) 13.30 (36.96) “ “ 2.16 Study Skills Total 17.81 (23.12) 1.54 (32.56) “ “ 6.86** Spelling 18.38 (29.16) 16.63 (39.07) “ “ .11 9 Reading Total -.22 (20.60) 11.31 (60.70) 49 58 .32 Language Total 22.67 (33.07) 12.67 (64.27) “ “ .21 Study Skills Total 13.56 (39.55) 13.81 (64.95) “ “ .00 10 Reading Total 1.57 (28.40) 27.83 (61.90) 34 54 2.37 Language Total 2.65 (31.58) 21.33 (73.04) “ “ 1.03 Study Skills Total -3.50 (34.30) 25.02 (73.05) “ “ 2.00 11 Reading Total 3.19 (21.41) 13.14 (61.11) 42 58 1.02 Language Total 5.95 (19.63) 5.81 (57.19) “ “ .00 Study Skills Total 4.45 (31.78) 20.74 (57.19) 2.80 7 Social Studies 7.32 (23.92) -7.03 (38.07) 68 40 5.79* 10 Math 34.00 (29.93) 12.50 (70.79) 32 54 1.88 11 Math -4.78 (16.79) 5.10 (50.03) 46 59 1.65 10 Science 21.98 (23.61) 19.32 (62.98) 89 54 .13 IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 73 Detailed Analysis of Student Impact Results Language Arts For students who used lesson plans from the Instructional Planner in an area of language arts and for the control comparison group significant main effects were found for group (i.e. IP Use vs. Control) for the grade 7 students in total reading scores and total study skills scores. On each of the tests, students in the control group scored significantly better than the IP group overall (F(1, 80) = 4.34, p < .05; F(1, 80) = 8.17, p < .01). There was no significant overall difference between the IP and control groups for language total scores. There also were no significant overall differences for any of the seventh grade students between pre and posttest scores. There was, however a significant interaction effect (group x test score) for the reading, language and study skills tests (F(1, 80) = 9.12, p < .01; F(1, 82) = 4.47, p < .05; F(1, 82) = 6.15, p < .05). In examining the interaction effect, Results indicate that although the control group had significantly higher scores on the 1999 test than the students who were to use the Instructional Planner lessons in the subsequent school year. By Spring 2000,students in the IP group made significant gains and were able to catch up to the control group in both reading and study skills. For students in the eighth grade who used language arts lesson units from the Instructional Planner, significant main effects were found for group on reading total, language scores, study skills and spelling scores. Unlike the seventh graders the overall scores of the IP group were significantly higher than the scores of the control group. Significant gains from pre to posttest scores also were found for all students (both IP and control) (F(1, 94) = 23.94, p < .01; F(1, 94) = 8.08, p < .05; F(1, 93) = 9.71, p < .01). No significant interaction effects were found, however, indicating that while both 8th grade groups scored better on the 2000 test, the students using Instructional Planner units whose achievement scores with higher initially were able to maintain their edge over the control group. Ninth, tenth, and eleventh grade students using Instructional Planner lesson units also had reading, language, and study skills test scores that were significantly higher (p < .05) than students in the control group overall. This is not surprising given that students in the treatment pool from these grades consisted of students in honors language arts classes. The two exceptions were the grade 10 students total reading scores and study skills scores. Here, the Instructional Planner students did not score significantly higher than the control group on the test. There also were no significant main effects for the 1999 vs. 2000 scores. All students in the 9th, 10th and 11th grade Instructional Planner and control groups showed no significant gains in total reading, total language and study skills standardized scale scores. There also were no significant interaction effects. The results for grades 9,10, and 11 in Language Arts suggest that students in both the Instructional Planner usage group and control groups did not significantly improve their IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 74  scores over the course of the year. Students in the Instructional Planner group, however, did maintain their significant edge over students in the control group. In summary, results for students who used language arts units from the Instructional Planner indicate that for students whose scores fell significantly below the control group in 1999, use of these units allowed them to catch up with their peers. Although Instructional Planner students whose 1999 scores were significantly higher did not significantly increase their scores, they were able to maintain their lead over their peers. Mathematics, Science and Social Studies Students who used Instructional Planner lesson units in science also scored significantly higher than the control group students on the Science sub-test of the Stanford Achievement Test (F(1, 141) = 41.46, p < .01). Both students who had Instructional Planner lessons and control group students had significantly higher scores on the 2000 test than on the 1999 test (F(1, 141) = 31.16, p < .01). Here again a non-significant interaction effect suggests that students who used Instructional Planner lesson units improved their scores from 1999 to 2000 and maintained their lead over the control group students. A similar result was found for tenth and eleventh grade students who used mathematics units from Instructional Planner. Students in both grades 10 and 11 had total mathematics scores that were significantly higher than the students in the control group (F(1, 74) = 101.39 p < .01; F(1, 103) = 15.16, p < .01). All students in the two tenth grade groups, Instructional Planner and control, also showed significant gains, F(1, 74) = 8.79, p < .01, from the 1999 to the 2000 test. There were, however, no significant gains for either the instructional planner or control groups in the eleventh grade. No interaction effects were found for either tenth or eleventh grade students suggesting that the mathematics scores of the tenth grade students improved overall and students using the Instructional Planner units maintained their lead over the control group students. Eleventh grade students did not show mathematics gains from 1999 to 2000 but again they maintained their edge over the control group students. Students who used social studies Instructional Planner lesson plans had social studies sub-test scores that were not significantly different from the control group overall. Across both the Instructional Planner group and the control, no significant gains from 1999 to the 2000 test were demonstrated. Unlike the results for science, and mathematics, however, there was a significant interaction effect between group and the 1999 and 2000 test scores, F(1, 106) = 5.79, p < .05. Scores of students using Instructional Planner lesson plans were the same as those of students in the control group in 1999. The students in the Instructional Planner group, however, showed significant gains from 1999 to the 2000 test while the students in the control group did not. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 75  Summary of Results on Student Impact Taken together these results indicate that use of Instructional Planner lesson units helps students who are under-performing make substantial gains in a relatively short period of time (within a single school year). For students already performing above the average, use of Instructional Planner units maintains their performance lead. This study demonstrates these results across grade levels and across curriculum areas. It should be noted that on any given version of the Stanford Nines, there are only a few items, at best, that directly relate to the IGOs covered by a lesson unit from the Instructional Planner. Being able to see any significant impact on such a broad measure from such focused instructional strategies is unusual since the test is not an ideal measure for use in detecting such specific changes. The limit of impact with higher performing students to maintaining their lead, while an important finding since the expectation would normally be a regression towards the mean, may be a ceiling effect of the test itself since there are not enough items on the specific IGOs to demonstrate much range in growth. This study examined impact on students at various grade levels and in the main instructional areas with the evidence supporting active use of lesson units from Instructional Planner as a strategy to improve and maintain higher levels of student achievement in core academic areas. Future investigations should include teacher variables such as experience level, role as both developer and user of IP units vs. just user, and other teacher variables to further understand differences in relative impact of the Instructional Planner and to verify these findings with larger sample sizes. Professional Development Impact from Instructional Planner • Participating teachers recognize the value and significance of the professional development process they went through even though they almost uniformly reported it as one of the most difficult professional experiences they have engaged in. It is significant to note that teachers in this Reinventing program were able to value and appreciate their experience while simultaneously discussing its rigorous nature. The participating teachers uniformly agreed that there was a discrepancy between what they considered to be a lesson plan and what the program required them to produce as a Reinventing lesson plan. As such, they felt that they were creating entire units rather than lessons, and that the time required to produce these units was somewhat unrealistic and overwhelming. As one teacher commented, "I'm not sure any of us realized what we were getting into.... We didn't realize that we were to develop the project as it went along. We thought there would be something more concrete rather than us being the innovators.” In spite of this, the majority of teachers also reported that this does not diminish either the professional or IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 76 personal value of the experience. Teachers clearly indicate that their ability to use technology and to integrate technology and standards-based materials into their teaching has drastically increased. As one teacher explained, "Creating the lesson plans was difficult. It was hard work, but the pay off is undeniable. Students learn and think they are playing. I would (and do) recommend Reinvent to anyone!" Moreover, a number of teachers who participated in the Reinventing Education program have taken on leadership roles around professional development and materials dissemination. As discussed in greater detail below, a number of teachers have continued to be a part of the program, moving from positions of trainee to ones that involve being trainer, facilitator and developer. Classroom Practice and Technology Literacy • The Reinventing Education program in West Virginia has resulted in the creation of rich Web-based resources for teachers that specifically integrate the state’s Instructional Goals and Objectives (IGO's). Teachers in this program were successful in completing the lesson plans required of them. All of these lessons have gone through a rigorous jurying process that ensures that they are closely tied to state standards, and the large majority of the lessons have been approved and posted in the Best Practices section of the Wired for Learning Web site. Combined with the summer presentations around the state, these lessons should now have widespread utility beyond the core group of teachers who created them. In turn they are serving as models for continued lesson development. • Reinventing Education teachers have reported that they are becoming more student-centered in their teaching practices, particularly with regard to the use of technology. Though many of the teachers who applied to participate in this program were already considered innovators in their schools, they commonly reported their experience with the program has led to a transformation in how they organize and manage their classes. In terms of practice, most of the teachers interviewed and surveyed stated that, at a minimum, they were actively integrating technology into their classrooms in ways and to a degree that they were not previously doing. Specifically, teachers indicated that their increased familiarity with the Internet and its resources has made it possible for them to allow their students to explore topics on their own more deeply and openly, and that this in turn has enabled them to increasingly allow students to drive their own learning. A direct result of this, teachers report, is that their students are more engaged and excited about the teaching and learning that they are experiencing. On another level, however, some teachers explained that the professional development process that they experienced as they created lessons around Internet resources, and as these plans were critiqued and revised, has helped them to think differently about the power dynamics within the classroom and their own role as teacher. For instance, because these teachers were given only a small number of computers for their students, all Reinventing materials were required to include multiple activities to be used simultaneously. The purpose of this was to make sure IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 77 that every student was engaged in an activity at any given time, even though they would not all be able to do the same thing, i.e., be on the computers all at once. This sort of student-centered classroom organization was not being used by a number of these teachers. While at first intimidating, these teachers have come to value that their students are now learning in a more independent environment not focused solely on a single figure in front of the classroom. For instance, one teacher explained that, "Five years ago I would never have seen myself teaching this way.... Now [I'm] individualizing instruction based on student needs, rather than whole-group instruction. Students work on projects that have real-life experiences. For the first time I see kids excited about math." • Preliminary findings suggest that there may be a correlation between teachers' use of Reinventing Education materials and increases in student scores on standardized state tests. Initial analyses of student test score data indicate that students who were in classrooms of Reinventing teachers who were high users of program materials during the 1998-99 school year had greater gains in test scores than students in the classrooms of low materials use. These findings are limited to date in mathematics and to a very small sample.65 We consider this pattern to be an extremely encouraging sign of a programmatic impact on student learning that should be investigated more deeply as teachers become increasingly comfortable integrating Reinventing materials into their classes. Teacher reports of their observations about the learning that takes place among their students while using program lessons, and about the increased motivation and engagement they are seeing in their classes, further extend this quantitative information. As one teacher wrote on her end of year survey, "I strongly believe that my students are more attentive and more motivated due to the use of technology. The dreaded book report is now exciting because students culminate the exercise with a PowerPoint. Further, the benefit of communicating with students in Australia or Virginia is overwhelming. Regular exposure to computers has made the practice a daily methodology instead of something 'extra.' We truly are preparing students for the future through the Reinvent lessons and related technologies." Technology Integration in Core Curriculum Areas • Teachers who participated in the program have reported an increase in their skills and confidence in their own use of technology and in engaging their students with technology in their classrooms. It is well documented that educators generally need to get beyond a certain threshold of comfort with computers and the Internet, and acquire a certain level of confidence in their skills, before they will begin introducing these tools into their classrooms. Many of the teachers who have participated in the Reinventing program had very little experience using Internet- 65 The research team analyzed test score data for the final cohort of participating teachers, which includes the other three curriculum areas. Next year we will continue this work, examining whether or not this pattern exists for teachers who use juried lessons developed by other teachers but who have not been through the Reinventing lesson development process themselves. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 78  based technologies in their classrooms prior to their involvement in this program. In interviews with and surveys of teachers in all four Reinventing disciplines, there was a clear pattern of growing confidence around using technology themselves and facilitating students and other teachers in its use, which was not present when they first entered the program. Teachers in the program are using technology more often, changing how they teach by making their instruction more student-centered and standards-based, and clearly feeling more at ease and capable in their use of these tools, even beyond the specific applications that were used in the program. As one teacher commented, "I have learned an immense amount of material while involved with Reinvent. I am now comfortable teaching others, including an adult basic computer skills class, a student summer enrichment computer camp and staff development in a variety of computer technology applications.” • Nearly all teacher participants report that they are continuing to implement program materials in their classrooms on an ongoing basis. Though the extent of materials use and integration varies within the group, a large majority of Reinventing teachers indicated that they are continuing to use the materials they developed through the program in their classrooms as well as plans created by other teachers as well. While there is not any current expectation that these teachers continue to develop Reinventing lessons, some teachers have said that they either are creating, or plan to create additional Internet-based lessons that are modeled after those created through Reinventing Education program. Evidence of Institutionalization Though there were numerous challenges encountered throughout the implementation of the Reinventing Education program in West Virginia, the project has not only met the majority of its goals but also has been successful in acquiring continued support from both within and outside of the State Department of Education. The following are specific areas where we have found evidence that the project is becoming institutionalized within the state. • The WV Reinventing Education program has been able to leverage the lessons learned during its pilot phase, successfully expanding the program to greater numbers of teachers. This project has, over its three years, enlisted three separate cohorts, each with slightly different roles and degrees of success. Though the second cohort, which was an expansion of the first group of math teachers, faced numerous challenges as it increased the scale of participation, the program was able to incorporate lessons from that experience as it rolled-out into its third group of participants. Thus the first two groups can both be seen as pilots used to hone the solution implementation for the program's final expansion, and there is strong evidence that the project was able to examine and adjust each successive phase to ensure that its goals and expectations were being met. One important lesson pertained specifically to the tools themselves. Throughout the program the technology-based tools that were integral to the development process IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 79 were not in place, or were only in beta stages, when they were required. This led to a loss of time and work on the part of teachers and program management. There were a number of occasions when program leadership planned to use tools that ultimately were not delivered, requiring changes in planning and eliciting frustration among the participants. More significantly, there were also a number of instances when the tools participants were relying on had been insufficiently tested and ultimately failed during use by the teachers. At these times, a great amount of participant work was lost and time was wasted, all of which made it difficult for the program management to smoothly carry out its work. A number of times these glitches resulted in the loss of work that had to be reproduced by participants. The program manager has become very resourceful in dealing with these problems, and has learned to put backup mechanisms in place to ensure that work does not get lost and time is used as efficiently as possible. However, many of the teachers indicated that they were not adequately prepared for the R&D nature of their involvement in the solution implementation. • Teachers from the math groups and the CTRC (social studies, language arts and science) group have maintained participation and taken on new roles and responsibilities as the program has expanded and evolved. As Science, Social Studies and Language Arts teachers were brought into the Reinventing project, a number of teachers from the earlier math group took on responsibility around training and supporting. Similarly, as the program has implemented its Professional Development I and II models, 36 teachers from all of these disciplines have moved into leadership roles that involve the implementation of multiple workshops around the use of Reinventing Education materials. • Through the Professional Development I (PD I) model, the Reinventing Education program has introduced its resources and practices to well over a thousand teachers in schools across the state. According to the program leadership, to date, 11 Reinventing teachers have gone on to participate in the PD I program as workshop facilitators and trainers. These teachers have conducted a total of 14 workshop series with a combined total of 173 teacher participants; 147 of these participants have in turn carried out staff development efforts in their own schools to a combined total of 1470 school staff members. In addition to these PD I courses, there have been approximately 20 general Reinventing Education presentations given around the state during the summer of 2000 as part of the program’s information dissemination efforts.66 • The program has been successful in attaining continued support from the State, as well as additional outside funding, as it has broadened its scope and vision in an effort to have an impact on more teachers at various levels within the education system. For instance, the materials, processes and models developed through the Reinventing Education program were incorporated into a PT3 grant in an effort to bring these technology rich resources and professional development models 66 activities of PDI teachers. These numbers were supplied by the program management, who is responsible for tracking the workshop IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 80 to both elementary and higher education as part of an effort to change the training of new teachers. Ongoing Challenges As our evaluation of the Reinventing Education initiative continues, there are several challenges that are still faced and that we will use to guide our assessment in West Virginia; they are: • Facilitating teachers who have not gone through the lesson development process to apply the pedagogical strategies supported by the program. As the West Virginia Reinventing program takes on a new form, we suggest that thought be put into understanding where the transformative power lay and how to incorporate that experience in this latest iteration of the program. What kinds of professional development experiences, for instance, are needed before teachers internalize the pedagogy built into Reinventing lessons if they have not themselves gone through the lesson development process? Do all teachers need to create lessons using the system developed in this solution in order to get the same benefits as the initial groups of teachers, which were in part due to the opportunity to work with Instructional Goals and Objectives while creating lessons and to receive constructive feedback through the jury panel. The emerging practice of using assessment data to determine needs before selecting lessons or even creating new ones may need to continue to be part of the refined process. • Gauging whether or not identified student weaknesses are changing as a result of teacher participation in the program. The relationship between any one set of professional development experiences and student learning is complicated. While initial findings have suggested that students being taught by teachers who frequently use the juried lessons, which target specific areas of need, show improvement on standardized tests, these findings are based on a small sample of teachers who have benefited from participating in an intensive lesson development experience. The stage is now set in West Virginia to more fully understand how this involvement is having an impact on student achievement through further research that focuses on the triangulation among the program and its materials, teacher practice and student work. • Understanding how the West Virginia Reinventing Education professional development process might be changing teacher practices in the classroom by going beyond teacher self-reporting. This project has been extremely successful in leading teachers in four disciplines through the process of developing technology-rich resources. As an outcome of this process, these teachers are now comfortable users of computers and the Internet and have the skills to continue to integrate them into their teaching beyond the use of these early Reinventing lessons. The lessons in the Best Practices database are strong public evidence of content reform around technology. This kind of content reform is an early phase in the longer trajectory of pedagogical reform more generally. While there is some early evidence of pedagogical reform as well, this evidence is mostly through self-reporting. We now need to add indicators of IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 81 changed teaching practice that more directly draw on the specific changes taking place within actual classrooms. • Teachers felt that the structure of the materials they were expected to develop was unnecessarily cumbersome and redundant and that some of the workshops should have been differently structured and better organized. Both the math and the scale-up teachers have repeatedly reported feeling that the structure of the lessons, and the online interface used for entering them, needed revising. One of the consequences of this is that, when asked if they are continuing to create Reinventing lessons, they often point out that there is no need to go through such a rigorous structure and process if it is no longer required of them — though they might create “Reinventing-like” lessons that use Internet resources in a similar manner. Also, because teacher schedules are so constrained by time they are often extremely sensitive as to whether the professional development efforts they participate in are run in an efficient and productive manner. In this regard, a number of Reinventing teachers indicated that the support they felt was truly necessary could not be given high enough priority because of unnecessary processes built into the program. For example, too much of the process of making, sharing, and jurying good lessons seemed superfluous to the teachers. While the program had a rationale for the structure being imposed on the lesson plans, it was not communicated to the teachers in a manner that helped them make sense of its utility. Similarly, there was not enough flexibility on the part of the program for it to reconsider what it might define as "a good lesson" by incorporating teachers' feedback into this definition. An example of teacher comments around this issue are: "The Wired for Learning site is far too difficult for the average teacher to navigate. Imagine: it takes a three-hour graduate class to learn to use it! The lessons are too ‘heavy’ for the majority of teachers. Quick, simple-to-use lessons would have been more beneficial, even for me, and I developed several of them," and "Having survived the training and developing of lessons, I am delighted to have the knowledge, resources, contacts, and lessons. However, the process was much more complicated and time-consuming than necessary. We were subjected to hours of advise/training that was not important or needed and we floundered in areas where more help was needed. The multiple peer online checklists, which had to be zipped and unzipped, were the most frustrating part of the process. Teachers do not have time to waste. Much of our time in the training sessions could have been more productive. In such training sessions, teachers should help to plan the agenda. I appreciate what I have learned and what I'm able to provide to my students as a result of this grant." • The state needs to identify ways to encourage teachers to use the online component of the program to communicate with one another for professional purposes, and to visit the site in an ongoing manner to utilize the various tools within Wired for Learning. Most of the teachers reported that they are using the Wired for Learning Web site to some extent, but only a few are visiting it on a regular basis. For instance, though a number of teachers indicated that they are maintaining teacher homepages, almost none of them had done group projects. Additionally, a IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 82 number of teachers indicated that they are not using Internet technologies to maintain ongoing professional communications with other Reinventing teachers, though they do maintain personal relationships and use email to keep up personal contact with some of their colleagues. As the program has moved on to PD I and PD II, however, use of the Web site is becoming a more integral part of this work. • Internet-based resources are dynamic and timely, and therefore must be continually monitored and updated over time. Though teachers have been asked to check that hyperlinks in their lessons are current, this may not be a sustainable means for maintaining materials. Some teachers are knowledgeable of the tools that are available that automatically check links and flag those no longer working, and commented that they are unsure why these are not being used to inform them when a link is broken. Again this often gets framed by the participants, especially the ones that have gone through the pains of working on a system that is still in development, as bewilderment or frustration that the project tools, and the procedures for using them, are sometimes being implemented in ways that unnecessarily make tasks more difficult for overloaded classroom teachers rather than less difficult or more efficient. • Much of the success of the project has depended on the work of a single person, who has had only limited support. Given the investment in and expansion of this program state-wide, the lack of a staff beyond the project manager is becoming a bottleneck and a liability. The longer term viability of the program and its ability to survive continued growth or personnel change without developing the necessary human infrastructure is questionable. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 83 Appendix 1: Watch-me!-Read Evaluation IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 84 What students learn with Watch Me! Read Susan M. Williams Vanderbilt University September 14, 2000 Watch Me! Read (WM!R) is a computer-based program for emerging readers developed by researchers at IBM’s T.J. Watson Research Center. Initial tests of this software were conducted from 1994-1997 in the Philadelphia Public Schools through a grant from IBM’s Reinventing Education Program. In 1997, the Reinventing Education Program announced a new grant to the Houston Independent School District (HISD) to replicate and scale up the Philadelphia test of WM!R. This report summarizes a study of the effects of using Watch Me! Read (WM!R) on first grade students' motivation, comprehension, and word recognition. The study was conducted at Will Rogers Elementary in Houston, Texas during the 1999-2000 school year. WM!R is being used primarily as a center activity at Will Rogers. For approximately 1 hour each day, groups of 4-5 students work independently at assigned activities located at stations around the room. One of these stations contains 3 computers where students use WM!R. In other stations, students engage in art, science, or math activities; read or listen to books on tape; or write stories while their teacher works with individuals or small groups. The teachers report that WM!R is always the first choice of activities. “When I announce the groups that are in centers and I tell them, ‘Group #1 is at computers,’ then that group cheers, ‘Yes, yes, YES, yes!’" One of the striking things about students at the computer station is their intense level of engagement. As one of the Will Rogers teachers puts it, “Basically they love the computers and they will go to that first if they are given a choice. And I do feel that they are more motivated to read while they’re on the computer, because they are talking to the computer and it’s very interactive, whereas when they’re having to read independently ...they are more likely to be off task compared to how they act when they’re at the computer.” The goal of the study reported here is to understand what students learn from their reading activities during center time—specifically to compare what they learn while using WM!R with what they learn while reading without assistance. The study focuses on students’ engagement in the reading task, their comprehension of what they read, and their ability to recognize words from stories they read. The following paragraphs first describe the students participating in the study, the testing procedure, and the materials used. Then the results are presented and summarized. Students IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 85 Participants in the study were students in Will Rogers' three heterogeneous first grade classrooms. All 34 students who returned signed permission letters from their parents were tested individually outside the classroom. Prior to the December test, students were classified as low, medium, and high ability readers based on the judgement of their teachers. An equal number of students at each ability level from each class were assigned to each interviewer and to each of the stories in order to counterbalance possible effects of these variables. By the time of the May interviews, students had improved greatly in their ability to read independently. Although exact data have not been compiled at this time, it is estimated that approximately 80% of the students were reading on a second grade level or higher in May compared with 20% in December. Procedure The testing for each student lasted approximately 1 hour. Students were asked to practice reading two stories, one on the computer and another in a small paper booklet the same size and layout as the WM!R display. Each student was asked to choose whether to practice with the computer or the paper booklet first. Twenty-six out of thirty-four students chose to practice first with the computer. Students who chose the computer were asked to practice reading out loud for five minutes using WM!R. During this time, they were left alone while the experimenter went to another part of the room. A video camera recorded their words and actions. At the end of the 5 minutes, the experimenter returned and asked them to retell the story they had just read without looking at the computer. If they did not identify the main characters, the setting, the main character's problem and it's solution, they were explicitly prompted for this information. Then they were then given a paper printout of the story and asked to reread only the part of the story they had completed on the computer. This process was then repeated using the paper booklet and a different story, i.e., students practiced reading the booklet out loud for 5 minutes while the experimenter left the vicinity. Then they were asked to retell and reread the part of the story they had read during practice. Students who chose to read the paper booklet first did the same activities in reverse order. Two interviews were conducted following this procedure, the first in early December 1999 and the second in mid-May 2000. Materials IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 86 The stories used in this study are part of an assessment system developed at Vanderbilt University for use with first grade students. Each story is at a level appropriate for students at the beginning of second grade. The stories are 375-400 words long and have no illustrations. Stories at this level were chosen to provide materials that would challenge all students. The assignment of books was counterbalanced so that half of the students at each ability level read Story A on the computer and Story B in the booklet and the other half of the students read Story B on the computer and Story A in the booklet. A different pair of books from the same series and at the same reading level was used in the May tests. Scoring of data Videotapes of students practicing, retelling, and rereading the stories are the primary data source for this study. The three variables of interest were task engagement during the practice session, and accuracy of retelling and rereading the story following practice. The data were scored as follows: Task engagement. Students’ engagement in the reading task was used as a measure of motivation. Engagement was scored on a four-point scale with 4 being intensely involved in reading. A score of 4 means that the student never looks away from the booklet or computer screen, he or she is reading out loud as instructed, and reading the story from beginning to end. A score of 1 means that the student looks away often, gets out of their chair, flips pages or presses keys on the computer randomly. A score of 1 indicates that there is no evidence that the student is reading. Ten samples from the 5-minute practice session were scored using this 4-point scale. The samples were 5 seconds in length and were spaced at 30-second intervals throughout the session. Story retelling. A reader’s ability to retell a story accurately and completely is often used as measure of comprehension. Students’ retellings were transcribed from the videotape. A list of idea units – characters, setting, events and other information from the story was compiled by dividing a transcript of the story into words and phrases. Students’ retellings were matched against this list and a percentage score was calculated. Credit was given for idea units at any level of importance. Students who did not recall the main character, setting, problem, and resolution of the story were specifically prompted for this information, e.g., “Was this story about an animal or a person? What was their name?” Credit was given for idea units elicited by these prompts. Retelling scores were based on only the portion of the story actually read by the students. Word recognition. Students were asked to reread only the part of the story that they were able to complete in the five-minute practice session. The word recognition IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 87  scores represent the percentage of words they recognized during this rereading. Words that were skipped or proper names were not included in this calculation. Data analysis. For each of the three variables of interest, the scores were analyzed using a 2 x 2 (Time of Test x Media) analysis of variance with Time of Test (December or May) and Media (Computer or Booklet) as repeated measures factors. Results Task engagement. Students are intensely engaged when using WM!R and this engagement does not falter even when they no longer need the feedback that the software provides.1 Although engagement while reading the paper booklet increased from December to May, it remained significantly less than engagement with WM!R. The details of task engagement data and analyses are presented in Fig. 1 and in the following paragraphs. Analysis of task engagement scores indicates significant main effects of Time of Test, F (1,26) = 6.76, p = .015; and Media, F (1,26) = 19.77, p < .001. There was a significant Time of Test by Media interaction, F (1,26) = 18.64, p < .001. Simple effects analysis of Media at each Time of test reveals that students are significantly more engaged when using WM!R than when reading the booklet. This effect was true in both December, F (1,26) = 22.72, p < .001; and in May, F (1,26) = 6.15, p = .020. 1 In May approximately 80% of the students were reading at second level or above. It was assumed that they could read most of the test materials without assistance. Preliminary analyses using individual reading levels estimated by the teacher indicate an interaction between reading level and students' level of engagement. This interaction appears to be due primarily to increased interest in reading the paper booklets for high-ability students rather than a lack of interest in WM!R. (Final analysis of student engagement using more precise assessments of students' reading levels are underway at this time.) IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 88  Fig. 1. The effect of media on students' engagement in an independent reading task Word recognition. Students are able to recognize significantly more words after practicing stories using WM!R than after practicing stories independently with a paper booklet. (Recall that students first had an opportunity to practice reading the stories using either the booklet or the computer. Then, these materials were put away and the same passage was reread on a typed sheet of paper.) This effect appears to be due to the feedback provided by WM!R in contrast with the absence of assistance when reading independently. It may also be due to the intense engagement while using WM!R. Not only was performance more accurate with WM!R in December, it remained more accurate in May even though students had improved greatly in their reading ability. The details of word recognition data and analyses are presented in Fig. 2 and in the following paragraphs. Analysis of task engagement scores indicates significant main effects of Time of Test, F (1,10) = 34.92, p < .001; and Media, F (1,19) = 10.84, p = .004. There was no interaction between Time of Test and Media. Preliminary analyses of the December data using teacher's estimated reading level for individual students indicate that there is a main effect of reading level, i.e., students at higher reading levels recognize a greater percentage of words when asked to reread a practice passage. Interestingly, there was no interaction of type of media with reading level, i.e., word recognition accuracy for all reading levels is higher when rereading passages originally read using WM!R than when rereading passages originally read independently in a paper booklet. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 89  Fig. 2. Mean word recognition scores following reading practice with WM!R and paper media Story recall. Students recalled a significantly greater percentage of idea units from stories read using WM!R than from an equally difficult story read independently from a paper booklet This was true for both the December and May tests. The details of story recall data and analyses are presented in Fig. 3 and in the following paragraphs. Analysis of story recall scores indicates a significant main effect of Media, F (1,26) = 22.03, p < .001. There was no main effect of Time of Test or interaction between Time of Test and Media. Preliminary analyses of the December data using teacher's estimated reading level for individual students indicate that there is a main effect of reading level, i.e., students at higher reading levels remember a greater percentage of idea units when asked to retell a practice passage. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 90 Interestingly, there was no interaction of type of media with reading level, i.e., retelling percentages for all reading levels are higher when retelling passages originally read using WM!R than when retelling passages originally read independently in a paper booklet. Fig. 3. Mean retelling scores following reading practice with WM!R and paper media Summary and conclusions This study provides strong evidence of the benefits of using WM!R to support independent reading practice. Students using WM!R are more highly engaged in reading than when reading without assistance. They are able to read more challenging material. Their comprehension is significantly better than when they are reading alone. When asked to reread, their word recognition is significantly better. These effects are large and consistent. Analyses of the effect of students’ reading level on their engagement, comprehension, and word recognition while using WM!R are not complete at this time. Preliminary analysis of the December data with estimated reading levels indicate that even students who can decode and read the test passage fluently score significantly better on tests of word recognition and comprehension when using WM!R. IBM's Reinventing Education Initiative Implementation Report March 27, 2001 Page 91