iWalkThrough Technology – An Interview with David Ruff of the Great Schools Partnership

Today we offer an interview with David Ruff of the Great School Partnership, a nonprofit organization supporting the Senator George J. Mitchell Scholarship Research Institute, regarding the development of the iWalkthrough™ system. The Partnership “works to redesign and strengthen public and private education to improve the quality of learning for all students” and to help “shape America’s finest secondary schools.”

Ruff is the co-Executive Director of the Great Schools Partnership and has been a major player in the development of the cutting-edge tool. As is our custom at OpenEducation, we post our interview in question and answer format.

According to the web site, “The iWalkthrough™ system was developed by the Great Schools Partnership to help educators continually improve instructional quality.” Can you give some insight as to the intent of the iWalkThrough tool from your perspective?

All of the things we look at with the iWalkthrough have been tied to research that leads to increased student achievement. To be clear, all items that could tie to positive achievement are not listed – that is a non-exhaustive list.

But everything we do have here has clearly been tied to student achievement. Our goal is to help teachers be able to hone their instructional practices. Getting the end data is not the goal for us – getting the data to help teachers improve their instructional practice is. The end goal is better instruction in the classroom because we know that better instructional practices will lead to higher levels of student achievement. We believe that teachers want feedback about their practice, that they want to get better at it. They want the data to be presented in a way that is helpful, not hurtful, but they want feedback. This is about bettering teacher practice and teachers having control over bettering their practice. Fortunately, the vast majority of teachers want to get better – this hasn’t been a hard sell.

Can you give me a brief overview of those various pull down menus that are available to observers and what data is collected?

The first few pull down windows feature basic demographics, the school, the date of the observation, the content area, the grade level, and the class size. For class size it is the number of students present not the number scheduled to be there. If a school wants, they can also identify the teacher but that is up to the school.

This demographic data is very important – if we want to know about science classes we can disaggregate the data – if we want to know about ninth grade, we can disaggregate the data – if we want to know what time of the year the observation occurred, was it in October or was it in January, we can disaggregate.

From there we look at the pedagogical aspects. We look at what the teacher is doing and we look at what the student is doing. We look at Bloom’s Taxonomy and the levels of Bloom’s and we look at levels of student engagement. Then we have a series of learning approaches that are picked up – these are approaches we know lead to higher achievement levels. For example, we look at: Are students using technology? Are teachers using technology? Are there discrete learning tasks, this is one way to look at differentiation, and we look at is there student work on the walls?

We have a lot of data on our schools, but our teachers don’t necessarily have a lot of data, and we don’t have much data on our instructional practices. What data we do have on instructional practice is mostly anecdotal, teachers talking to other teachers about the things that they do. What’s important in our minds is for teachers to get a sense of the data. We want them to have ownership, to manipulate it themselves so that they can understand it. In other words, how do they need to see the data to be able to make sense of it themselves?

We wanted to get away from a data collection system that was one, collected by just a few people, and two, controlled by a few number of people. By being web based and with all teachers having passwords, they can all go in and see the data as they need to see it and to cut it as they need to cut it.

Again, according to the web site, the iWalkthrough™ system was “developed by a team of experienced educators in collaboration with hundreds of schools and teachers, and field tested in thousands of classrooms.” Can you give some more details as to this process, who was involved, and what types of testing took place?

When we first got started, we as a staff brainstormed all the things we might see – we thought about, “what are the things we would see if we went into your classroom?” We came up with a pretty exhaustive list. Then we went through it and made our first cut at it by saying what are the things we could actually see if we went into a classroom, We had a lot of things on the list that if you interviewed the teacher or if you talked to the student you could get at the information but we wanted the things an observer could see. We wanted the process to have great credibility and great reliability so we did not want the process to depend on someone having to ask the right question to get the key data. That cut our list easily in half.

We then took the items that remained and asked ourselves which of these can we prove are tied to research that demonstrates increased student achievement. Not the practices we liked – not the practices we thought were good – but the practices we can concretely prove through research lead to increased student achievement.

One small, but clear example is the category regarding student work being displayed. If kids can see examples of high quality student work, what the target is and what to aim for, then the quality of their work will increase. It made sense to us that work had to be recent work but we could not find that in the research. So what we look for is student work in the content area being displayed. Even though we might feel that the work should not be 6 months old we could not find any research that said that it mattered how old the displayed work was.

That helped us cut our list further. We then began working with schools and administrators. Our coaches took this out to the schools and asked, “What do you think of this? Are we doing the right things? We were talking with about 25 different schools, faculties and administrators to get feedback from them. We took all of that feedback – not all of it correlated, some people said that’s great, some said that’s awful – but through that feedback we came to what we thought were the wisest choices and we shortened the list further.

The first time we used the tool was actually in a way that we do not use it much anymore. We visited every school in the Great Maine School’s Partnership. We took a team of from 4 to 12 people depending on the size of the school and we did a number of walkthroughs. It was fascinating – we were not web based at that time and everything was going through my hard drive on that small lap top. At the end of the day we synched them all up, sent the data to the printer, made copies and gave them to the faculty.

When the faculty saw it was when we first saw it. There was none of this let us manipulate it, let us do this to it, let us bring this piece out. They saw it when we saw it and I will tell you the credibility that we gained from that was tremendous. It showed us the value of teachers being able to see the data right off the bat. Unfortunately so much of the data we share with teachers has been cleaned in some way.

We began in the fall and we were doing the walkthroughs in the spring so the entire process, from initial conception and brainstorming to testing was about 7-8 months. By the next fall we were able to go web based.

Did you make any further changes after that initial set of steps?

Yes, the refinement has been ongoing. We have made a number of changes based on further feedback from teachers. As one example, the categories for engagement began as quartiles, 0-25%, 26-50%, 51-75%, and 76-100%. The feedback we got and the experience that we gained was that teachers first off said anything less than 50% really doesn’t matter, that’s bad. So we made that change. Then teachers said that 75% isn’t really all that high a bar. They also wondered what if it was close to 100%, it was a big range from 76 to 100, so that was why we went to a new cut, 91-100%. That came from the use of the data by teachers.

The actual process of walking into a classroom, doing the observation, then stepping into the hall to make their notes before moving to the next classroom, all that has really remained relatively the same. What has changed are the categories – as teachers have discussed and cut the data and as they have discussed what the data meant for their instruction they began asking how to see the day in different ways. It was those discussions that led to the category changes and other refinements.

As I understand it there are four different tools (high schools, middle schools, elementary schools, and athletic programs). Can you give a brief overview of the differences in the tool, especially the software categories used, as it relates to each of these four distinct levels? Can you briefly go over the various pull down menu options and how those might change depending on the age level or whether the device is used for athletic purposes?

The high school, the middle school, and the elementary school, we call them the academic side, they are all very similar. In fact, the high school and middle school are almost identical. At the elementary level we have categories that relate to literacy strategies, numeracy strategies, and connections to prior learning strategies. Those are not currently in the high school version though I predict that within a year or two we are going to start seeing them at the high school level as well.

The fourth unit is athletics. We are very excited about this. At the end of the day, when you think about it, what does an athletic director do? The AD makes sure that everyone is at practice then walks around observing those practices. Athletic directors have been doing walkthroughs for years! They just didn’t have any way to collect the data. At the end of the year, when an athletic director sat down with a coach to discuss the season the conversation was not always about the art of coaching. It was often about what a tough loss that was, who went here, who went there as opposed to what is your practice as a coach. We have three pilot sites, Maine Central Institute, Kennebunk High School, and Bowdoin College. People are really excited about it – we will be looking to roll that out this fall, right now it is still on individual hard drvies.

Can you give a basic overview of the technology utilized. My understanding is that things are web based as opposed to being hosted on a local school server. Can you give a brief explanation of how the technology works and the rationale for being web based?

Yes, as I mentioned before we went web based, that way we became wireless. That takes one component out of the process – the data is up instantly. If it was on a hard drive somewhere you would need to get to that hard drive but if it is web based when the ninth grade team meets in a room they can have access to the data. At the same time if the social studies department is meeting across the hall they too have access to the data. The basic premise is better access for data input and data output, data can be input from anywhere and it can be accessed from anywhere.

Again, these folks have access to school data, to aggregate data. We suggest at the outset that schools not identify the teacher. But teachers are now asking, “What’s my data look like?” Roughly one third or so of our schools now identify the teacher – if the school does then the teacher also has access to his or her data. It is password protected, at meetings the data being looked at is aggregate.

Can you give a sense as to how many schools/administrators are currently using the device? Could you give a rough break out of how those numbers differ among the four different tools mentioned above (how many high school, how many middle school, etc.)?

We have 79 schools in the State of Maine and another eight in Oregon. The vast majority of those using the device are high schools. I would estimate, very roughly, that we have about a dozen or so elementary schools and another dozen maybe of middle schools. The remainder of those implementing are at the secondary level. Then we have our three schools piloting our iWalkthrough athletics.

Can you give a rough estimate of the costs associated with putting these devices into use in a school district (perhaps give a couple of different scenarios depending on the size of the district)? Please include all aspects, training, etc, and how the costs might break out.

A basic subscription for a school runs $600 followed by an individual charge of $25 per teacher in that respective building. That covers the cost of use, setting up the passwords, etc. Those dollar figures are for the pertinent software. School districts must purchase their own PDAs for the classroom observers, we don’t supply the PDAs. A real plus thus far has been that for current Maine school districts the $1500 training cost has been covered by grants secured by the Great Schools Partnership. That covers our staff costs for the day of training. That may change in the future.

Are there any planned upgrades or changes on the horizon for the device?

One of the things we will be doing with the new system coming out is to give the average response and high one, high two responses based on the data from all schools collectively. Therefore, if you are looking at say, engagement, where the categories are 0-50, 51-75, 76-90, and 91-100, the data will show first, here’s my school, then over the 20,000 or so observations total, here are how many are in the 91-100% engagement category. What’s the average? Then without specifying the school, we will be able to specify a school, again not giving the name, but there is a school that has say 68% of its teachers in the 91-100% engagement level. Then as a school, teachers can say here is where we are, and, here is where we can be.

When people look at the site come July they are going to find a lot more resources available online – data analysis protocols for example. And if a teacher wants resources regarding Bloom’s Taxonomy or engagement, we will have lists of materials on the site. Another aspect is to develop 6-8 individualized questions or categories to become part of the pull down menu, suites of questions, say about literacy for example, that can be selected by schools. There would be say six questions on literacy available should a school want to focus on that.We want them to be vetted by the organization, to pass the same muster as our original pull down menus. Schools will also have the option to develop their own questions, though they would not be able to have comparision data as those questions would be unique to that school. This is all a key aspect of our feedback and refinement processes.

Next up, we talk with several schools in the field implementing this new technology.