Research shows that "research-based" pedagogy doesn't work! (but discussion does!)

I. Excellent scholarship is multivalent

Great education scholarship does many things at once.  McKeown, Beck and Blake's excellent paper about content pedagogy vs. skills pedagogy is a good example: they manage to show that content pedagogy can actually do a better job of teaching skills than  a pedagogy that aims at direct instruction in those skills; but the paper also contains some remarkable data implying that what seems "natural" to teachers may be both unnatural and ineffective.

Another wonderful paper on reading is a 2006 classic by Martin Nystrand, called "Research on the Role of Classroom Discourse As It Affects Reading Comprehension".  Nystrand's paper, a magisterial review of 150 years of research into American classroom discourse, argues powerfully that discussion-based teaching can produce significant gains in reading comprehension.  The paper is worth reading for its main point, but it also sheds light on our era's focus on misguided reforms that claim to be "data-driven--and as a bonus, Nystrand is a sharp, funny writer.  In fact, I laughed aloud more than once while I was reading the paper last summer.  At one point, my wife asked what I was laughing at, and I tried to explain:

In his paper, Nystrand says that the National Research Council (the NRC) and the Department of Education have declared that education is woefully unfounded in data-based research, being stuck instead, according to these august bodies, in "a 'folk wisdom' of education based on the experience of human beings over the millennia in passing information and skills from one generation to the next." Denigrating "the experience of human beings over the millennia" is kind of funny--education is a deep human enterprise, and most of the deepest human experiences (love, child-rearing) have rightly held modern "data-based research" at a skeptical arm's length--but it was  Nystrand's next sentence that really cracked me up:

"While considerable recent work supports the NRC's contention that the education research base is little used in schools, research also strongly suggests that this conception of education research is, in the case of the pedagogical effects of classroom discourse, inappropriate, and even counterproductive." (393).

My wife didn't think it was funny.  I tried to explain: in other words, I said, research does show that education is not often based on research, but research also shows that education that is based on research is inappropriate and counterproductive.

Ha!

II. The limitations of prescripted lesson plans; the limitations of rigidly "scientific" social research

Nystrand's whole paper, like that sentence, is excellent, both witty and wise. In fact he isn't really saying that all research is counterproductive, only research of the type privileged by the National Research Council, that is, research in which the relevant variables are strictly defined and rigidly controlled.  For classroom practice to be strictly defined and rigidly controlled, it must follow prewritten scripts.  The use of prewritten scripts, however, precludes a more flexible pedagogy, and more particularly precludes rich conversation of the kind Nystrand calls "dialogic" (after Bakhtin and Volosinov, and informed by Vygotsky--what is it with Russians?).

I'm very interested in this" dialogic" mode of classroom discourse, and how it relates to what my colleagues and I are trying to do this year with our focus on close reading of brief passages, but first I had to think about the implications of Nystrand's point about research.  I have been curious and confounded by the inadequacy of a lot of "research-based" pedagogy, and Nystrand's point about education research gave me a new persective on why the research is so weak.

If a process is standardized, it is much easier to reliably measure results.  To produce a more scientifically rigorous and reliable conclusion, then, you want to compare one type of highly standardized instruction with another.  For instance, if you're comparing a skills approach to a content discussion approach, you don't want your data muddied by the way different teachers implement the two approaches.  If you have two teachers doing each approach, you want both skills teachers teaching the skills in exactly the same way, and you want the two content discussion teachers running the content discussions in exactly the same way.

However, Nystrand is arguing, standardization itself may be bad pedagogy.  In the example of a study comparing a skills approach to a content approach, neither approach might be as good as one that allows for more open-ended discussion.  Any script, he suggests, might be too limiting.

In some ways, this seems to support my suspicion of Common Core standards and indeed all backwards lesson planning in English class: if you know exactly where you are heading in a lesson from the very beginning, how can you be excited about the lesson, and how can you make it exciting?

III. The (Research-Proven) Benefits of Discussion-Based Pedagogy

His insight into the limitations of much education research is, however, basically an aside; Nystrand's paper is largely about the significant  benefits of discussion-based pedagogy. Ironically, he makes his case by citing dozens of empirical studies, including more than one showing that discussion leads to large gains in reading comprehension. He also says that the best discussions are those in which the teacher controls text and topic but students have both time and interpretive control.

According to Nystrand, the virtues of discussion-based pedagogy, rather than lecture or recitation, have been reiterated in American scholarship on teaching since at least 1860, when Morrison complained that "young teachers are very apt to confuse rapid-fire question and answer with effective teaching." One of the great virtues of Nystrand's paper is his review of the history of American classroom discourse.  According to Nystrand, it is well-documented that the prevalent form of classroom discourse for over a hundred years has been "recitation," that is, rapid-fire question and answer about the basic facts of what the students have already read.  Some writers celebrated this, but many argued that recitation was essentially pointless, and that discussion was a much better practice.  Nystrand quotes Bloom (1956) as saying that in his observation over 50% of instructional time in American classrooms was taken up with teachers talking.

This is still true, apparently.  In a helpful overview of the state of English instruction, Nystrand reports on research finding that 95% of ELA teachers value discussion, and their students report that discussion helps them understand their readings--yet only 33% of the teachers regularly make room for it.  Indeed, Nystrand writes, other researchers found that in the 58 ninth grade classes they observed, only five classes had any group work, and 90% of this group work was merely collaborative seatwork.  In an even more disturbing finding, the same study reported that in these 58  ninth grade classrooms, open-ended whole class discussion averaged only 15 seconds a day.

This is disturbing, but perhaps not particularly surprising.  Fostering "open-ended discussion" does not come naturally to many teachers, including me at times.  But it is incredibly important to do so, partly because a "discussion-based" classroom results in greater recall and comprehension, but also because discussion-based classrooms help build what Nystrand calls "classroom epistemology" and what my colleagues and I have been calling "habits of mind."

First, Nystrand's own research and the research of others, involving thousands of classroom observations, shows that the amount of classroom discussion is strongly correlated with improved recall and comprehension of the reading, and, interestingly, a greater response to the "aesthetic elements" of literature.  Second, a "dialogic" classroom also creates a culture of thinking more deeply and helps students develop an "identity" as a reader--which will indirectly help students over the longer term.  As Nystrand writes, citing dozens of studies, it is "the conversations teachers lead with their students" that define the way students think about literature.

IV. How we should put "dialogism" into practice in our classrooms

In short, Nystrand says that research shows that we should control text and task (and we should prefer problematic and difficult passages (duh!)), and we should give students time and interpretive responsibility.  According to Nystrand, a recent meta-analysis of 49 studies found that the most productive discussions were clearly framed by the teacher, but gave students extended time to elaborate their ideas and allowed for considerable flexibility in what students actually said. Nystrand also suggests that pair discussion is not as good as larger group discussions.

This fall, I am planning to put some of these ideas into practice.  In addition to high-volume independent reading, my ninth grade classes this year are going to be largely focused on brief passages.  In addition to writing short analytical essays about such passages, and partly to prepare them for the writing of the papers, they will have extended large-group discussions about brief passages more than once a week (as Nystrand suggests, I will mostly retain control of text and topic).  My hope is not only that these discussions will help their reading comprehension and their analytical abilities, but that the discussions will also foster a culture of thoughtful discourse and intellectual and aesthetic curiosity.  We'll see how it goes.

I was going to stop this section there--but then it occurred to me that "seeing how it goes" is far from a simple concept.  In fact, I wish I had a better handle on how to know whether changes I make in my teaching actually make any difference to student learning--but what I see over and over again is that even people who are very interested in data, like Bruce Baker of the excellent blog schoolfinance101, are very suspicious of the validity and reliability of evaluating teachers based on test scores.  Is my own classroom a large enough sample for me to have much confidence that any improvement in my students' performance is actually due to my own efforts?  Hm...  I'll try to find out.  In the meantime, let me go back to my conversation with my wife, after I laughed aloud at Nystrand's wit.

Coda:  Is research-based reform ever helpful?

I didn't get to talk about discussion-based pedagogy with my wife--and it actually took me a while to get back to the article itself, because after I laughed at Nystrand's sentence about research showing that education based on research is inappropriate and counterproductive, she asked a very good question: Well, when is research-based reform not counterproductive?

Um, maybe in automobile safety, I said.

My wife just looked at me.  It was a witty look, and I laughed, because I could see what she was thinking. Automobile safety is a perfect example to use, since decades of research and data-based improvements in safety have resulted in a system that still produces nearly 40,000 violent deaths in the United States every year.  It's possible, said her look, that what's needed is not evidence-based tinkering but a paradigm shift.

Okay, obviously I should have offered a medical example--knee-replacement surgery, maybe--in which research-based reform is useful.  Still, my wife's response was a wise one: what is necessary in the case of automobile safety, as well as what would be helpful in the case of reading, is to look, not more closely at the component parts of the system, but at the system overall.  This is in a way what Nystrand attempts to do in his article about classroom discourse, suggesting that we need to get the sage off the stage.  I wonder if we might also look at the overall system by looking at so called "natural experiments"--for instance by trying to look at the differences between the Finnish educational system and that in the US, or the automobile safety system in Holland as compared to that in the US, in something of the way a recent excellent post on the New York Times's economics blog discusses teacher salaries.  Another way of looking at the overall system is by comparing it to other systems--by comparing education to health care, say.  I hope to explore these comparisons a bit in future posts.