Is good data-driven decision-making possible?

Is data-driven decision-making possible in schools? I've long worried about it and am glad to hear Chris Lehman echo this concern in his excellent presention (excellent because I agree with almost everything he says?) at Ignite Philly:

Chris, you're singing the librarians' old songs about research and problem-based learning and presentation and authenticity. Great!

But back to data-driven decisions. Why are these nearly impossible to make well at a school level? From an earlier column, A Trick Question:

At last spring’s interviews for our new high school library media specialist, the stumper question was:

"How will you demonstrate that the library media program is having a positive impact on student achievement in the school?"

How did that nasty little question get in there with “Tell us a little about yourself” and “Describe a successful lesson you’ve taught”? Now those questions most of us could answer with one frontal lobe tied behind our cerebellums.

Given the increased emphasis on accountability and data-driven practices, it’s question all of us, librarians and technologists alike, need to be ready to answer - even if we are not looking for a new job or don’t want to be in the position of needing to look for one.

While I would never be quick enough to have said this without knowing the question was coming, I believe the best response to the question would be another question: “How does your school measure student achievement now?”

If the answer was simply, “Our school measures student achievement by standardized or state test scores,” I would then reply, “There is an empirical way of determining whether the library program is having an impact on such scores, but I don’t think you’d really want to run such a study. Here’s why:

• Are you willing to have a significant portion of your students (and teachers) go without library services and resources as part of a control group?
• Are you willing to wait 3-4 years for reliable longitudinal data?
• Are you willing to measure only those students who are here their entire educational careers?
• Are you willing to change nothing else in the school to eliminate all other factors that might influence test scores?
• Will the groups we analyze be large enough to be considered statistically significant?
• Are you willing to provide the statistical and research expertise needed to make the study valid?"

I surmised then that "No school I know of has the will to run such as study."

If test scores are the sole measure of "student achievement," there are indeed some things we in schools can be excellent at doing with data. We can identify individual students who perform below established norms and we can look at groups of students with certain characteristics (ELL, FRP, SpEd) and see how they compare with the norms. We can do trend tracking of such groups.

We are good at determining which groups need help. But what comes next is the "gotcha."

Schools are unwilling and unequipped to do controlled studies on the effectiveness of any single intervention over a period of time to improve the test scores of a school or group. The typical pattern is to throw as many changes into a curriculum as possible and hope something sticks.

Let's say our SpEd population is showing low reading scores. A building may well decide to:

1. Increase the use of differentiated instruction
2. Try a new computerized reading program
3. Increase the SSR program

Pretty good strategies, huh? But here is the rub. What happens, let's say, if the groups scores rise. Any one of these interventions may have been effective. All of them may have created some of the improvement. Two of them in combination may have led to the improvement. The Hawthorne Effect might be in play and gains this year, might not be shown next year. Some may be effective, but take more than a year to show results. A completely extraneous variable may have been present (a new teacher or principal, perhaps).

Schools should not be tasked with doing research. This was what university lab schools are (were) for. Every school doing its own research on effective educational practices makes no more sense than every hospital being a research hospital and every student being a guinea pig.

I am not sure what the answer to this problem might be nor has anyone to date given me a good solution to this problem (if even willing to admit a problem exists.)  I do believe that carefully applied valid research can help teachers improve their instructional practices.

It just shouldn't be up to the practitioner to also be a researcher.

Your thoughts?

To Put a Man on the Moon

I rarely praise my boss. It's not that he isn't an exceptionally good leader and manager, because he really is.  I just don't want to look like a suck up. 

When we met last week to discuss my departmental and professional goals, he made a simple request. He asked to attend our next technicians' meeting so he could explain our district's new strategic road map.

I appreciate this.

There is an apocryphal? story about a reporter doing interviews at NASA in the late 60s. He was interviewing different people about their jobs. The response of one custodian was interesting. When asked his job, he said, 'To help put a man on the moon."

If you asked members of your technical or library staff what their jobs were, how many would say, "To educate children"?

And if they wouldn't, why not?

Schools for the governors

Scott McLeod's last two posts (one, two) on Dangerously Irrelevant ask an important question: If students are to grow up understanding their constitutional rights, do they need to be able to practice them in school?

In 1992 Johnathan Kozol observed in his book Savage Equalities that the US has two kinds of schools: those for the governors and those for the governed. It's a pithy statement that neatly categorizes the kinds of schools we have in the US. His argument was based on the economic support shown for schools for the poor and those for the well-to-do, but I also think it applies to educational programming. Is your school helping create self-determined individuals or just rule followers?

I've long advocated for research questions that have a "action" element to them. This rubric's final indicator advances that:

A Research Question Rubric: not all research questions are created equal. (from Designing Research Projects Students (and Teachers) Love)

Level One:     My research is about a broad topic. I can complete the assignment by using a general reference source such as an encyclopedia. I have no personal questions about the topic.
Primary example: My research is about an animal.
Secondary example: My research is about the economy of Minnesota.

Level Two:     My research answers a question that helps me narrow the focus of my search. This question may mean that I need to go to various sources to gather enough information to get a reliable answer. The conclusion of the research will ask me to give a supported answer to the question.
Primary example: What methods has my animal developed to help it survive?
Secondary example: What role has manufacturing played in Minnesota’s economic development?

Level Three:     My research answers a question of personal relevance. To answer this question I may need to consult not just secondary sources such as magazines, newspapers, books or the Internet, but use primary sources of information such as original surveys, interviews, or source documents.
Primary example: What animal would be best for my family to adopt as a pet?
Secondary example: How can one best prepare for a career in manufacturing in the Twin Cities area?

Level Four:     My research answers a personal question about the topic, and contains information that may be of use to decision-makers as they make policy or distribute funds. The result of my research is a well support conclusion that contains a call for action on the part of an organization or government body. There will be a plan to distribute this information.
Primary example: How can our school help stop the growth in unwanted and abandoned animals in our community?
Secondary example: How might high schools change their curricula to meet the needs of students wanting a career in manufacturing in Minnesota?

I usually joke that Level Four is for the "over-achievers."

But I may need to rethink that glib comment...

What assignments empower your students, helping make them governors, not the governed?