Friday, September 14, 2012

Research shows that "research-based" pedagogy doesn't work! (but discussion does!)

I. Excellent scholarship is multivalent

Great education scholarship does many things at once.  McKeown, Beck and Blake's excellent paper about content pedagogy vs. skills pedagogy is a good example: they manage to show that content pedagogy can actually do a better job of teaching skills than  a pedagogy that aims at direct instruction in those skills; but the paper also contains some remarkable data implying that what seems "natural" to teachers may be both unnatural and ineffective.

Another wonderful paper on reading is a 2006 classic by Martin Nystrand, called "Research on the Role of Classroom Discourse As It Affects Reading Comprehension".  Nystrand's paper, a magisterial review of 150 years of research into American classroom discourse, argues powerfully that discussion-based teaching can produce significant gains in reading comprehension.  The paper is worth reading for its main point, but it also sheds light on our era's focus on misguided reforms that claim to be "data-driven--and as a bonus, Nystrand is a sharp, funny writer.  In fact, I laughed aloud more than once while I was reading the paper last summer.  At one point, my wife asked what I was laughing at, and I tried to explain:

In his paper, Nystrand says that the National Research Council (the NRC) and the Department of Education have declared that education is woefully unfounded in data-based research, being stuck instead, according to these august bodies, in "a 'folk wisdom' of education based on the experience of human beings over the millennia in passing information and skills from one generation to the next." Denigrating "the experience of human beings over the millennia" is kind of funny--education is a deep human enterprise, and most of the deepest human experiences (love, child-rearing) have rightly held modern "data-based research" at a skeptical arm's length--but it was  Nystrand's next sentence that really cracked me up:

"While considerable recent work supports the NRC's contention that the education research base is little used in schools, research also strongly suggests that this conception of education research is, in the case of the pedagogical effects of classroom discourse, inappropriate, and even counterproductive." (393).

My wife didn't think it was funny.  I tried to explain: in other words, I said, research does show that education is not often based on research, but research also shows that education that is based on research is inappropriate and counterproductive.


II. The limitations of prescripted lesson plans; the limitations of rigidly "scientific" social research

Nystrand's whole paper, like that sentence, is excellent, both witty and wise. In fact he isn't really saying that all research is counterproductive, only research of the type privileged by the National Research Council, that is, research in which the relevant variables are strictly defined and rigidly controlled.  For classroom practice to be strictly defined and rigidly controlled, it must follow prewritten scripts.  The use of prewritten scripts, however, precludes a more flexible pedagogy, and more particularly precludes rich conversation of the kind Nystrand calls "dialogic" (after Bakhtin and Volosinov, and informed by Vygotsky--what is it with Russians?).

I'm very interested in this" dialogic" mode of classroom discourse, and how it relates to what my colleagues and I are trying to do this year with our focus on close reading of brief passages, but first I had to think about the implications of Nystrand's point about research.  I have been curious and confounded by the inadequacy of a lot of "research-based" pedagogy, and Nystrand's point about education research gave me a new persective on why the research is so weak.

If a process is standardized, it is much easier to reliably measure results.  To produce a more scientifically rigorous and reliable conclusion, then, you want to compare one type of highly standardized instruction with another.  For instance, if you're comparing a skills approach to a content discussion approach, you don't want your data muddied by the way different teachers implement the two approaches.  If you have two teachers doing each approach, you want both skills teachers teaching the skills in exactly the same way, and you want the two content discussion teachers running the content discussions in exactly the same way.

However, Nystrand is arguing, standardization itself may be bad pedagogy.  In the example of a study comparing a skills approach to a content approach, neither approach might be as good as one that allows for more open-ended discussion.  Any script, he suggests, might be too limiting.

In some ways, this seems to support my suspicion of Common Core standards and indeed all backwards lesson planning in English class: if you know exactly where you are heading in a lesson from the very beginning, how can you be excited about the lesson, and how can you make it exciting?

III. The (Research-Proven) Benefits of Discussion-Based Pedagogy

His insight into the limitations of much education research is, however, basically an aside; Nystrand's paper is largely about the significant  benefits of discussion-based pedagogy. Ironically, he makes his case by citing dozens of empirical studies, including more than one showing that discussion leads to large gains in reading comprehension. He also says that the best discussions are those in which the teacher controls text and topic but students have both time and interpretive control.

According to Nystrand, the virtues of discussion-based pedagogy, rather than lecture or recitation, have been reiterated in American scholarship on teaching since at least 1860, when Morrison complained that "young teachers are very apt to confuse rapid-fire question and answer with effective teaching." One of the great virtues of Nystrand's paper is his review of the history of American classroom discourse.  According to Nystrand, it is well-documented that the prevalent form of classroom discourse for over a hundred years has been "recitation," that is, rapid-fire question and answer about the basic facts of what the students have already read.  Some writers celebrated this, but many argued that recitation was essentially pointless, and that discussion was a much better practice.  Nystrand quotes Bloom (1956) as saying that in his observation over 50% of instructional time in American classrooms was taken up with teachers talking.

This is still true, apparently.  In a helpful overview of the state of English instruction, Nystrand reports on research finding that 95% of ELA teachers value discussion, and their students report that discussion helps them understand their readings--yet only 33% of the teachers regularly make room for it.  Indeed, Nystrand writes, other researchers found that in the 58 ninth grade classes they observed, only five classes had any group work, and 90% of this group work was merely collaborative seatwork.  In an even more disturbing finding, the same study reported that in these 58  ninth grade classrooms, open-ended whole class discussion averaged only 15 seconds a day.

This is disturbing, but perhaps not particularly surprising.  Fostering "open-ended discussion" does not come naturally to many teachers, including me at times.  But it is incredibly important to do so, partly because a "discussion-based" classroom results in greater recall and comprehension, but also because discussion-based classrooms help build what Nystrand calls "classroom epistemology" and what my colleagues and I have been calling "habits of mind."

First, Nystrand's own research and the research of others, involving thousands of classroom observations, shows that the amount of classroom discussion is strongly correlated with improved recall and comprehension of the reading, and, interestingly, a greater response to the "aesthetic elements" of literature.  Second, a "dialogic" classroom also creates a culture of thinking more deeply and helps students develop an "identity" as a reader--which will indirectly help students over the longer term.  As Nystrand writes, citing dozens of studies, it is "the conversations teachers lead with their students" that define the way students think about literature.

IV. How we should put "dialogism" into practice in our classrooms

In short, Nystrand says that research shows that we should control text and task (and we should prefer problematic and difficult passages (duh!)), and we should give students time and interpretive responsibility.  According to Nystrand, a recent meta-analysis of 49 studies found that the most productive discussions were clearly framed by the teacher, but gave students extended time to elaborate their ideas and allowed for considerable flexibility in what students actually said. Nystrand also suggests that pair discussion is not as good as larger group discussions.

This fall, I am planning to put some of these ideas into practice.  In addition to high-volume independent reading, my ninth grade classes this year are going to be largely focused on brief passages.  In addition to writing short analytical essays about such passages, and partly to prepare them for the writing of the papers, they will have extended large-group discussions about brief passages more than once a week (as Nystrand suggests, I will mostly retain control of text and topic).  My hope is not only that these discussions will help their reading comprehension and their analytical abilities, but that the discussions will also foster a culture of thoughtful discourse and intellectual and aesthetic curiosity.  We'll see how it goes.

I was going to stop this section there--but then it occurred to me that "seeing how it goes" is far from a simple concept.  In fact, I wish I had a better handle on how to know whether changes I make in my teaching actually make any difference to student learning--but what I see over and over again is that even people who are very interested in data, like Bruce Baker of the excellent blog schoolfinance101, are very suspicious of the validity and reliability of evaluating teachers based on test scores.  Is my own classroom a large enough sample for me to have much confidence that any improvement in my students' performance is actually due to my own efforts?  Hm...  I'll try to find out.  In the meantime, let me go back to my conversation with my wife, after I laughed aloud at Nystrand's wit.

Coda:  Is research-based reform ever helpful?

I didn't get to talk about discussion-based pedagogy with my wife--and it actually took me a while to get back to the article itself, because after I laughed at Nystrand's sentence about research showing that education based on research is inappropriate and counterproductive, she asked a very good question: Well, when is research-based reform not counterproductive?

Um, maybe in automobile safety, I said.

My wife just looked at me.  It was a witty look, and I laughed, because I could see what she was thinking. Automobile safety is a perfect example to use, since decades of research and data-based improvements in safety have resulted in a system that still produces nearly 40,000 violent deaths in the United States every year.  It's possible, said her look, that what's needed is not evidence-based tinkering but a paradigm shift.

Okay, obviously I should have offered a medical example--knee-replacement surgery, maybe--in which research-based reform is useful.  Still, my wife's response was a wise one: what is necessary in the case of automobile safety, as well as what would be helpful in the case of reading, is to look, not more closely at the component parts of the system, but at the system overall.  This is in a way what Nystrand attempts to do in his article about classroom discourse, suggesting that we need to get the sage off the stage.  I wonder if we might also look at the overall system by looking at so called "natural experiments"--for instance by trying to look at the differences between the Finnish educational system and that in the US, or the automobile safety system in Holland as compared to that in the US, in something of the way a recent excellent post on the New York Times's economics blog discusses teacher salaries.  Another way of looking at the overall system is by comparing it to other systems--by comparing education to health care, say.  I hope to explore these comparisons a bit in future posts.


  1. Another thought-provoking post. I spent some time this summer taking a workshop to learn a new pedagogical spin that might interest you in light of what you've said here. The group (Right Question Institute) puts forth the premise that people only develop skills when they are applying them to achieve what they truly desire for themselves...which often means trying to answer their own questions. Thus (for example), we learn to be better researchers when we're trying to find the answer to a research question that matters to us. Similar (they would argue), we become better readers when we're reading for our own purposes (for example, we care about the characters or the topic or the plot...or whatever).

    Not surprisingly (this group's research indicates), students are hardly ever encouraged to ask their own questions in schools and are rarely taught how to formulate such questions. Instead, teachers provide the questions, which the students are then instructed to answer. Furthermore, students' questions rarely drive learning (and certainly not the larger curriculum). Thus (the group argues) students never learn how to drive their own learning and thus frequently lack the motivation to develop skills (because the learning of skills is disconnected from what should be the natural catalyst: students' own questions).

    The pedagogy the group recommends (see is incredibly simple, but I don't think it's simplistic. I'm trying it out this year. You might consider some variation of it as you structure the passage discussions in your classroom, for example.

    An interesting fact related to your post: the pedagogy this group advocates is NOT based on academic research (and they make a big deal about this) but instead is based on work the leaders did with working class parents in Lawrence and Lowell, parents who were struggling to find ways to formulate the questions they needed to ask in order to improve their interactions with school officials. This is experiential research, what was called "action-research" when I was in grad school. I find that I trust this stuff--and am far more willing to try this stuff--than the research that comes from those who are not intimately involved in the practices of day-to-day teaching.

    Thanks, again, for your post.

  2. MF: "An interesting fact related to your post: the pedagogy this group advocates is NOT based on academic research ... from those who are not intimately involved in the practices of day-to-day teaching."

    Yes, though I think one of the subtexts of Nystrand's article is that there are two kinds of academic research. One kind is the rigid, pre-planned experiment with a treatment group and a control group. This is the kind of research the NRC, the DOE and the reformers tend to privilege, and the kind Nystrand says is usually useless. A second kind of research is the kind that Nystrand and others seem to do, which involves a lot of observing of the teaching and learning that is already going on. It seems to me that the academics can serve the useful function of observer and reporter and aggregator of the good work that real teachers are actually doing.

    In any case, I'm going to look up the Right Question Institute.

  3. Thanks to both of you for your thoughtful posts. I'd like to hear more about, as I am trying something that seems sort of similar with my 10th graders.

    MF's comments about action-research reminded me of what I was taught in grad school: that in terms of education, there was quantitative research and qualitative research. I, having no interest or aptitude for the quantitative side, enrolled in a good course called qualitative research. We read books like The Good HIgh School and Among Schoolchildren and we went out to schools and observed and interviewed and wrote narratives about what we witnessed.

    At the time, it seemed to me that the qualitative research was as highly valued by the Academy (this was at Stanford) as the quantitative, and it was fully acknowledged that both types had limitations.

    Twenty years later, it seems no one talks about the qualitative side any more. I have a theory about why this is -- or at least what accelerated it in education, and it has to do with how schools of education have focused so much of their energy on inner city schools. Who can blame them? But I feel the net result is that the frameworks and solutions folks are developing in the research world tend to be frameworks and solutions that will/may help schools IN EMERGENCY, but are ill-suited to improving ALL schools. The quantitative approach works (somewhat) when it sees teaching as a skill -- it can help identify incompetent teaching and help schools reach some "passing" level. But if your schools and your teachers are already relatively strong, I think qualitative research is better at treating teaching as an art and pushing it to a higher level.

    This is a great blog. Keep going!

    1. Wow--that's a really interesting issue! I think your theory might be largely right--but I want first to pick up on the "may" half of your "will/may" formulation and resist the idea that schools in emergency SHOULD be heading in the direction of ever-more seatwork and test prep. One of the interesting things about Nystrand's article is that he reviews over a hundred years of observers of US schools and says that we have basically always been erring on the side of too much question-and-answer instruction and not enough real discussion. We've been, as a country, pushing a rigid and test-driven kind of teaching for about 10 years now, with no changes in reading scores to show for it. So while I do think there has recently been a big rhetorical focus on "failing" schools, I'm not sure the recent reform has done much to "help schools reach some passing level"--mainly, of course, because the problems in high-poverty schools stem from the poverty, not the schools and the teachers.

      That said, I'm not ready to give up on the quantitative side of things. A lot of the research Nystrand discusses is a combination, with lots of observation but also a lot of looking at test scores and seeing how they correlate with what was observed.

      I had an interesting exchange with an Ed. school professor a couple of years ago about these issues--maybe I'll tell that story in a post sometime soon.

  4. The stuff is interesting--I might try it with a couple of my classes this year. I also like it that their office is about two blocks from my house...

  5. What I always wonder is why it takes "modern" research to tell us how the great minds of history have always learned...engaged in dialogue, if not person-to-person then in conversation with texts. Heck, why even bother with great minds...we grow by talking about MORE stuff--by stretching linguistic experience...this can happen for anyone at anytime.

    removing the act of human responding to an utterance simply reduces the opportunity for the brain to form meaningful connections and then "randomly" respond.

    also, any chance you can send on the pdf to that article for those of us without JSTOR access?

    We are "researching" our way to managerial models that cohere only if we consider ourselves ant colonies.

  6. Those people follow these ideas who work on data entry thanks for share it how to create a database in excel.