How do we demonstrate our impact on student achievement?

Greetings from Golden, Colorado, where in about an hour I'll be working with educators from the Jefferson County Schools. One of the things I've been asked to visit with school librarians about is how we can demonstrate school library media centers' impact on student learning. I hear this one a lot lately.

In a rather off-handed manner, I dealt with the topic last summer in the blog entry "A Trick Question." But I realize I did not do the topic justice. So for the last couple weeks, I've been trying to distill some thought and readings on the subject. Find the results below. I am heavily indebted to Joyce Valenza, David Loertcher, Ross Todd and others who I cite in the little article below.

Normally, I would ask this at the end of the blog entry, but I am going to do it now: Please, if you have ideas about how YOU demonstrate your influence on your school, please let me and other readers of this blog know. This is not a done deal, but something we will need to continue to work on for the remainder of our careers. Thanks!

 

Demonstrating Our Impact: Putting Numbers in Context

One of my favorite quotes comes from George Bernard Shaw: We should all be obliged to appear before a board every five years and justify our existence...on pain of liquidation. While Shaw was commenting one’s social worth, his words today could come from any number of administrators, school boards and legislatures and be aimed directly at school library media specialists. Finding an answer to the question “How do we demonstrate our impact on student achievement?” is increasingly important for every library media specialist in the country to be able to persuasively answer.

I have long been frustrated with this question, especially when those asking want empirical rather than anecdotal evidence to support claims of effectiveness. To me, genuine empirical evidence is the result of a controlled study and no school has the ability or will to do a controlled study on library effectiveness. Would your school:

• Be willing to have a significant portion of its students (and teachers) go without library services and resources as part of a control group?
• Be willing to wait three to four years for reliable longitudinal data?
• Be willing to change nothing else in the school to eliminate all other factors that might influence test scores?
• Be willing to find ways to factor out demographic data that may influence test results?
• Be able to analyze a large enough sample to be considered statistically significant?
• Be willing to provide the statistical and research expertise and manpower needed to make the study valid?"
  

I know mine wouldn’t participate in such a study, no matter how clear-cut the evidence produced. So how do we demonstrate out impact using “numbers?”  Let’s look at a number of ways, none perfect, but when used in combination, powerful.

1. Standards and checklists. A common means of assessing a school library media program (and by inference assessing its impact on student learning) is by comparing an individual library media program to a state or national set of program standards. AASL’s Planning Guide for Information Power: Building Partnerships for Learning with School Library Media Program Assessment Rubric for the 21st Century (ALA, 1999) is one example of a tool that can be used to do such a comparison. Many states also have standards that can be used to evaluate library media programs. Minnesota’s, for example, can be found at <www.memoweb.org/htmlfiles/linkseffectiveslmp.html>.

Both AASL and MEMO use rubrics that quickly allow a media specialists to evaluate their programs. For example, MEMO’s “Standard One” under the Learning and Teaching section reads: “Is the program fully integrated?” and gives these levels

Minimum
25-50% of classes use the media program’s materials and services the equivalent of at least once each semester.
Standard
50%-100% of classes use the media program’s materials and services the equivalent of at least once each semester. The media specialist is a regular member of curriculum teams. All media skills are taught through content-based projects.
Exemplary
50%-100% of classes use the media program’s materials and services the equivalent of at least twice each semester. Information literacy skills are an articulated component of a majority of content area curricula.

While standards can and should be used to help evaluate a program, the direct link between meeting such standards and local student achievement is not present. While backed by research, best practices, and the experience of the standards writers who are usually experts in the field, these tools can only suggest what may make a local program more effective, not demonstrate that the current program is having an impact. While important, standards are guides, not evidence.

2. Research studies. The Colorado studies are a good example of using statistical regression analysis to look for correlations between variables. In the case of statewide library studies, the relationship of effective library programs and standardized test scores is examined. School Libraries Work, (Scholastic, 2006) is an excellent summary of this types of research. <www.scholastic.com/librarians/printables/downloads/ slw_2006.pdf>. These can and should be shared with principals. Some statisticians do not approve of regression analyses because they show correlation, not causation, and because it is very difficult to factor out other variables that may have impacted the correlation.

Other formal individual research studies and meta-studies are also worth sharing with administrators. Stephen Krashen’s Power of Reading, 2nd edition, persuasively stacks up a large number of individual research reports to demonstrate that voluntary free reading can improve student reading ability. And he concludes that when students have access to a wide range of reading resources (in libraries, of course), they do more independent reading.

Unfortunately, just as all politics are local, so are all assessments local. While decision-makers are usually quite willing to read and acknowledge studies done “elsewhere,” most still want to know the direct impact their local program is having.

3. Counting things. Year-end reports that include circulation statistics, library usage, and collection size data are a common way for building library programs to demonstrate the degree to which they are being used, and by inference, having an impact on the educational program in the school.

Ontario Library Association’s  Teacher Librarian Toolkit for Evidence Based Practice <accessola.com/osla/toolkit/home.html> contains a number of forms that can be used to track circulation and incidences of collaboration. Jacquie Henry provides a tool for tracking media center usage in the January 2006 issue of Library Media Connection.

Our district’s “Year End Report” asks library media specialists to enumerate the following:

Circulation statistics:
Number of print materials circulated
AV materials circulated
In-library circulation of print
In-library circulation AV materials
AV equipment circulated

Use of space:
Classes held/hosted
Drop in users
Computer lab
After hours
Other uses

Collections:
Number of books acquired and deleted
Number of AV materials acquired and deleted
Number of software programs acquired and deleted

Leadership team activities: (List any building/district committees on which you have served and your role on them.)

Instructional activities:
For primary, please list for each grade level library units taught that support classroom units and major skills taught.
For secondary, please list all units taught collaboratively and skills for which you had major responsibility for teaching.

Special programs or activities: (in-services, reading promotions, authors, events)
Please share a minimum of three instructional highlights for the past year. This is very helpful when concrete examples of media/tech services are needed.

Communications: (Please list how you have communicated with parents, staff and students this year.)

There is a movement away from counting things: materials, circulation, online resource uses, website hits, individual student visits, whole class visits and special activities conducted (tech fairs, reading promotions, etc.) to enumerating how many instructional activities were accomplished:  booktalks given, skill lessons taught, teacher in-services provided, pathfinders/bibliographies created and collaborative units conducted. Administrators are less concerned about how many materials are available and more concerned about how they are being used.

Information and technology literacy skill attainment, if assessed and reported, is another means of “counting” one’s impact. Our elementary library media specialists have primary responsibility for teaching these skills and complete sections of student progress reports similar to those done in math and reading. At the building level, it is possible for the library media specialist to make a statement like: “89% of 6th grade students have demonstrated mastery of the district’s information literacy benchmarked skills.”

4. Asking people. Asking library users to complete surveys and participate in focus groups are frequently used to collect information about the impact of the library media programs.

Some sources of surveys:
•    Johnson, What Gets Measured Gets Done (Tools): <www.doug-johnson.com/wgm/wgm.html>
•    McGriff, Preddy, and Harvey, Program Perception <www.nobl.k12.in.us/media/NorthMedia/lms/data/ percept/percept.htm>
•    Valenza, Power Tools Recharged (ALA, 2004)

Surveys of both students and teachers can be done either at the project level at completion, or on an annual basis. Joyce Valenza conducts video “exit interviews” of graduating seniors at her high school.

Survey-based data gathering was a powerful tool used by Todd and Kulthau to conduct Student Learning through Ohio School Libraries: The Ohio Research Study <www.oelma.org/studentlearning> in 2003. This type of study would be relatively easy to recreate at the building level.

5. Anecdotal data. Is there value to anecdotal evidence and stories? Despite my favorite statistics teacher’s dictum that the plural of anecdote is not data, I believe empirical evidence without stories is ineffective. One skill all great salespeople have is the ability to tell compelling personal tales that illustrate the points they wish to make. It’s one thing for the guy down at the Ford dealership to show a potential buyer a Consumer Reports study. But the real closer tells the story of how Ms. Jones buys this exact model every other year and swears each one is the best car she has ever owned. When selling (advocating for) our programs, our visions, and ourselves to those we wish to influence, we need to tell our stories. See “Once Upon a Time,” Library Media Connection, February 2002. <www.doug-johnson.com/dougwri/storytelling.html>.

Don’t discount how powerful “digital storytelling” can be as well. A short video or even photographs of students using the library media center for a variety of activities can be persuasive. How many times have you said, “If only the parents could see this, they would support the library 100%”? Though digital photography and a presentation to the PTA or Kiwanis organization, they can see your program.

Context and Focus Numbers alone, of course, mean little. They need to interpreted and placed in some type of meaningful context. Context can be achieved by setting and meeting goals and by looking at numbers in a historical context. Look, for example, at how each statement gets more powerful:

• 28 teachers participated in collaborative units (Is this good or bad?)
• 78% of teachers in the building participated in collaborative units (This tells me more.)
• 78% of teachers, up from 62% of teachers last year, participated in collaborative teaching units. (This shows a program that is getting stronger.)

It’s clear that a wide variety of means exist to assess a wide variety of library program activities. How does one choose the “what” and “how” of program evaluation?

David Loertscher’s Project Achievement <www.davidvl.org/achieve.html> suggests that data collection should be done at three levels in order to triangulate evidence: at the Learner Level; at the Teaching Unit Level; and at the Organization Level and provides tools to do just that. He also suggests evaluating the impact of the library program on four areas: Reading, Collaborative Planning, Information Literacy and Technology.

My suggestion is to pay careful attention to your building and district goals and annual objectives. If reading is a focus, then look at reading activities, promotions, collection development and circulation. (If there is a focus on a particular demographic within your school (ESL students for example), check to see if your circulation system will allow you to sort by that identifier.  You own goals and the accomplishment of them, can also provide an effective means of assessment.

We can no longer afford to complete a program evaluation once every five years and have the results thrown in a drawer and never used. Our assessments need to help us improve our practice, to serve as indicators for our planning efforts, and to be an integral part of our communication efforts with our teachers, administrators, parents and communities. Assesment, of course, takes time. But less time than finding another job.

How are you “demonstrating your impact on student achievement?”

Revisiting Stone Soup

From Scott McLeod's Dangerously Irrelevant blog entry Does technology impact student learning?

The Metiri Group has just issued a report commissioned by Cisco Systems [consider the funding source - Doug] that finds that technology can have significant impacts on student learning outcomes when implemented with fidelity. The report, Technology in Schools: What the Research Says, notes that school organizations must pay "serious attention . . . to leadership development, professional development for teachers, school culture, curricular redesign, and teacher preparation" in order to see the learning benefits of their technology investments. The results are no big surprise for technology advocates, but it is nice to see additional research support for our claims.

Scott, I'd go further than to say "...no big surprise." I'd say, "Well, duh!" (But then you are a nice person and I am not always.) Some of us indeed figured this out many moons ago. A story written long ago...

 Stone Soup: A Classroom Parable

Originally published in MultiMedia Schools, Nov/Dec 1996

    When Ms. Eastman returned to school one fall she found a large boulder had appeared in the middle of her classroom. It was about the size of washing machine, gray with silver glints, and could neither be ignored nor moved.
    “I think you’ll just have to make the best of it,” the principal advised, unable to explain how or why it came to be there. “I think the Board may have wanted it, and the budget just won’t support hiring a professional rock mover,” he said in a single breath, and resumed his telephone conversation with the parent of child who had just eaten a wall-mounted pencil sharpener.

***
    When Ms. Eastman’s students returned, they immediately pounced on the rock.
     “Hey, it’s like the one in my backyard.”
    “Check it out. It’s a throne.”
    “Nah, a bomber.”
    “It’s perfect for my desk!”
    “Where did it come from, why is it here, who gets to sit at it, why can’t we all have one?”
    Ms. Eastman asked the class to get to work, and as they opened their textbooks and began to quietly read, she distributed worksheets.

***
    The rock proved to be an annoyance. It was right in the middle of the room. It made creating a seating chart difficult. It was too far from the front of the room to be used as a stand for the overhead projector, and the irregularity of its surface made it a poor desk or surface on which to affix papers. On occasion, when a student had done exceptionally well, Ms. Eastman allowed that person to “Read on the Rock.” But for the first quarter, the entire class just worked around the boulder.
    Late one November afternoon, Ms. Eastman overheard two students engaged in a heated argument over the composition of the stone. “Settle this intelligently,” she admonished. “Both of you, go to the library, do your research, and come back and report to the class.” In less than half an hour, two excited children returned.  “It’s definitely basalt with quartz flecks. Hey, do know where this came from? A volcano...” The class listened intently as the pair shared their findings about the rock.

***
    Over her second margarita that evening, Ms. Eastman reflected on that afternoon’s class, and decided it had gone particularly well. The kids were enthusiastic and attentive. One of the two children who did the research on the rock performed at higher level and showed more interest in school than Ms. Eastman thought it possible for him to. She began to see the rock’s glitter in a new light.
    The next Monday morning, Ms. Eastman read the class the myth of Sisyphus, and asked the students to use questions raised during the discussion as the basis of their journal writing. Over the next week or so, articles related to rocks keep popping up in magazines and newspapers. Ms. Eastman used these as springboards for lessons in math and science and history. Soon students were finding and sharing information they themselves had found about rocks in their reading and viewing.
    Just before turning off the classroom lights one evening, Ms. Eastman caught a glimpse of white near the base of the stone. It was a note left in a crevice of the rock. When asked, the class sheepishly admitted the rock was serving as a classroom post office. “Are these the same kids I can’t get to put two cogent words together in their journals?” wondered Ms. Eastman. She struck a deal with the class: they could continue to write their letters as long as they revised one letter each week.  That letter would be read for grammar and spelling, and could be shared with the rest of the class.

***
    As the year progressed, many activities began to center around the rock. Parts of the rock easily broke away into pebble size pieces and the class began a business selling “Stone Soup Starters.” In the processes, students applied math, designed advertisements, and worried about ethics. At various times the rock was the setting for plays about the Pilgrims and pioneers on the Oregon trail. Science class divided into small groups which used the rock to demonstrate principles of acids and bases, friction, gravity, and sundials.  One morning a small wooden door appeared firmly attached to the base the rock, which one student adamantly declared was a passage to Van Allsburg Land. Van Allsburg Land soon had a language with its own syntax, a codified set of laws, and even its own culture - all which in some strange way reflected the world the class lived in and was trying hard to understand.

***
    One day in early May the principal called Ms Eastman to his office. “Do you have the correct code to the photocopier?” he asked.  Ms. Eastman looked puzzled. “You have made almost no photocopies since October. We need these numbers to satisfy the central office, you realize.” He was right, Ms. Eastman mused. She hadn’t used many worksheets, she’d used only pieces of her text book, and had not shown a videotape from beginning to end since November. Her grade book had only a few entries, but each of her students had a pizza box crammed with exemplary reports, graphs, drawings, and models she had asked them to collect since the last parent teacher conference.
    She reflected her class this year must have just been made up of exceptional children, just the right chemistry. They cooperated, they were genuinely interested in school, they held good discussions, and they were conscientious about the quality of their work. They seemed to be reading more newspapers and magazines, came back from the library with more materials and fewer complaints from the librarian, and could apply math principles more quickly and accurately to every day problems than previous years’ classes. Ms. Eastman was still in happy amazement as her graduate classes started in June.

***
    When Ms. Eastman returned to her classroom the following fall she found a computer had appeared in the middle of her classroom...

***

Have a wonderful weekend. I promise to use a part of it to read the Metiri report.

Paddle Your Own Canoe

Love many, trust a few, and always paddle your own canoe. Bumper Sticker

I decided long ago that I couldn't afford to wait for our administrators to take formal training in educational technologies. I needed tech-savvy administrators.

So without additional dollars, grants, projects, books, or meetings, these are some of the things our department has done to help "train" administrators:

• Set examples of good communication, planning and record keeping using technology.
• Involve our administrators in all technology staff development activities.
• Provide fast technical support and individualized training.
• Provide clear teacher and student information literacy and technology competency lists.
• Serve on building/district leadership teams.
• Serve as CIO to all administrators, finding and forwarding information of interest.
• Help administrators understand what they need to know.

Under the last one, we developed CODE77 rubrics for administrators even before ISTE came out with NETS-A. (The most recent - 2002 - version links correlates the CODE77 to the NETS-A standards.)

I think the majority of our principals and other administrators use and understand educational technology pretty well. These folks tend to be neither technophobes nor technophiles, which is just fine with me.  

I share the concern that without administrative knowledge and support, technology will not be used well in schools. But we can't wait for the magic wand, a new generation of administrators, or "requirements" from some higher power. We have to "paddle our own canoe" and work daily on the job to develop administrative proficiencies.  

How do you "teach" your administrator?