Saturday, February 23, 2013

For vocabulary, input volume matters

A recent issue of Edweek has another lame article purporting to connect research to the Common Core--again, most of the "research" is incredibly weak and the experts say silly things.  Nevertheless, the article does mention one classic piece of research, and looking back at that study leads, as usual, to the conclusion that reading volume matters a lot.

Unnecessary new research...
According to the first paragraph of the article:

     Children who enter kindergarten with a small vocabulary don't get taught enough words—
     particularly, sophisticated academic words—to close the gap, according to the latest in a
     series of studies by Michigan early-learning experts.

Well, duh.  Anyone who had looked carefully at the vocabulary research, or indeed simply thought logically about the matter, would know that no scientific studies are necessary to conclude that kids with small vocabularies can't possibly be "taught" enough words to close the gap, since the only truly significant way kids learn words is through reading them and hearing them used.  I have never yet read a study purporting to show that any class of students, anywhere, has been "taught" enough words to make a significant increase in their vocabularies.  The main study discussed in the article found limited vocabulary instruction across the board, and less instruction in "academically challenging words" at high-poverty-schools.  Neither of these findings is necessarily significant, because vocabulary instruction just doesn't make much of a difference.

...and humorless experts!
Throughout the article, ostensible experts are quoted saying silly things.  For instance, one scholar says that Kindergarteners should be taught academic words like "predict."  That might be reasonable, but then she goes on, "Why would you choose to emphasize the word 'platypus'? It makes no sense."  Hm.  What makes no sense to me is that someone who can't imagine a reason to emphasize a really interesting, cool, loveable word like "platypus" would have anything to do with children's education, let alone be on the faculty at the University of Michigan.

What we should be thinking about
The article spends a fair amount of space, and a cool decision tree sidebar graphic, on which words to teach.  Thinking a lot about this is probably a waste of time, since teaching words doesn't make much of a difference, except, perhaps, insofar as kids enjoy them.  What then should we be thinking about?  Well, the only decent piece of research cited by the article is the classic 1995 study by Hart and Risley that reveals the remarkable disparities in the numbers of words heard by kids from different socioeconomic backgrounds.  Upper-middle class kids hear 11 million words per year, while poor kids hear 3 million; by the age of three, upper-middle class kids know twice as many words.  The  two "key conclusions" of the Hart and Risley study are the following:

• The most important aspect of children’s language experience is quantity.
• The most important aspect to evaluate in child care settings for very young children is the amount of talk actually going on, moment by moment, between children and their caregivers.

These conclusions do NOT say that we should be spending our time deciding which words to "emphasize" or "prioritize" or teach; instead, what matters is how many words kids are hearing or reading.  In fact, I see no reason not to transfer the second conclusion of the Hart and Risley study to schools of older children, too, with only slight modifications. As children get older, we need to add reading to our model, since as we get older reading is essential for experiencing high volumes of sophisticated language, and the quality of the input may become more important, since you do want them to hear or read words thast they don't actually know, but the quantity is still much more important than the Edweek article acknowledges.  So I would extrapolate thus:

• The most important aspect to evaluate in child-care settings for older children (i.e., schools) is the amount of sophisticated language actually experienced by the children, whether from a caregiver (i.e. teacher) or by reading.

Of course, I suspect reading is probably more important than teacher-talk. The Hart and Risley study focused on talk among families, which is primarily one-on-one, and the best way to simulate that in a classroom with a student-teacher ratio of at least 20:1 is by having each child reading a book. So I'll conclude where I always do, with another form of my usual hypothesis:

• The most important aspect to evaluate in child-care settings for older children (i.e., schools) is the amount of reading actually going on.

Friday, February 15, 2013

Poverty matters in Wisconsin: No poor school achieves as well as any rich school

Corporate ed reformers like Joel Klein and Michelle Rhee always argue, as in their "manifesto" on "How To Fix Our Schools," that poverty can be overcome by schools and teachers.  That manifesto asks us to start with what they call "the basics" of what they call "failing schools":

     "the single most important factor determining whether students succeed in school is not
     the color of their skin or their ZIP code or even their parents’ income—it is the quality
     of their teacher.”

This is, of course, a straight-out lie.  Unfortunately, lies this convenient are zombies that need to be disproved over and over again. I recently saw, on the blog of a retired math teacher, a graph that (re-)does the job quite neatly.

The graph, gratefully reproduced below, shows very clearly the relationship between poverty and educational achievement.

The state of Wisconsin makes available a lot of data about all of its schools, and the retired math teacher blogger has made a chart comparing the average "student achievement score" for a school (a composite of the math and reading scores on the state tests for all students in a school) with the % of students who are eligible for free or reduced-price lunch.  Each blue dot represents a school; its vertical position represents student achievement, while its left-right position represents the % of "economically disadvantaged students."  (The curved lines are the first and third-degree functions that fit the data best--you can ignore them if you like.)

wisconsin school overall student ach score by pct of poor kids

What I like about this graph is its beautiful symmetry. Look at the top left hand corner of the graph, and you will see that there is not a single rich school (i.e. less than 10% of students eligible for free or reduced price lunch) in the entire state of Wisconsin whose "student achievement" the state measures at less than 54.  Similarly, there is not a single poor school (i.e. FRPL>90%) whose student achievement is above 53. In other words, there is ZERO overlap between the rich schools and the poor schools.

This graph, like the compelling ones compiled by UT physicist Michael Marder, shows that in the real world there is no school--not a single one!--that has succeeded in educating its poor kids to the level of even the lowest-achieving rich-kid schools.  It may be theoretically possible, but as far as I can tell it has never been done, and to say, as Klein, Rhee and their ilk do, that "the single most important factor" determining success in school is not parental income but teachers is offensive to teachers, to kids, and to anyone who cares about truth, science, data, or reality.

Wednesday, February 13, 2013

State of the (Educational) Union

In his State of the Union address last night, President Obama hardly mentioned K-12 education.  I didn't mind, partly because I think the President's inclination to support increased standardized testing and teacher evaluations based on test scores are misguided, but also because other issues he discussed would have far more effect on educational outcomes than any school reform.  In his speech, the President discussed health care reform and his proposed increase in the minimum wage, two initiatives that could make a significant difference in how well US children can read, write and think.

The President argued for Obamacare and for a higher minimum wage in both humanitarian and economic terms.  This is appropriate, since the US spends nearly twice as much on health care as every other industrialized nation and even raising the mimimum wage to $9, as the President proposed, would only bring it back to where it was over thirty years ago, and would boost the economy, which is hampered primarily by lack of demand, through increasing the income of those who spend everything they make.  At the same time, however, I wish he had mentioned their educational benefits, since a good case can be made that these initatives are more important for education than any testing regime, curriculum reform, or teacher evaluation system.

While schools and teachers are essential, most of the variance in student achievement is determined by factors outside the school building (that's why education reformers always say that teachers are "the most important in-school factor"), and health and poverty matter more than any others.  If Obama can succeed in providing health care for all and a higher, inflation-indexed minimum wage, not only will he have made our country fairer and stronger, he will be, for once, a true "Education President."

Monday, February 11, 2013

Conversations and discussions as (formative) Reading Assessments

After spending a few days thinking about little besides snow, I want to take an hour and try to write about reading assessment, and in particular about the kind of natural assessment that can happen in a conversation about a book.

All assessment is formative, and all assessment is also instruction
The only valid educational reason for assessment is to improve education; therefore, anything we think of as "summative" should probably be re-engineered to make it more formative. In addition, assessment is always, inevitably, teaching something, and we should consider, before we assess, what our assessment will teach.  But that's abstract...

Conversation and discussion as assessment
One good way to assess student reading is through conversation and discussion.  In a discussion about a text she has read, the student should get a lot of truly immediate feedback about her reading. 

While my students are reading in class, I go around and talk to individual students.  Sometimes these conversations are on a specific topic common to the whole class: in our grammar unit, for instance, I will have quick check-ins with kids about, say, prepositional phrases on the page they are reading; if I've assigned the class an essay about point of view, we will talk about that.  At other times, however, the conversations are totally individualized, organic exchanges in the style (I think) of Nancie Atwell, about particular issues in the books they are reading on their own, and sometimes about particular passages from these books (I usually stick to the passage they happen to be on just then).

Usually I have not read these books--and if I have read them, I don't remember the details all that well--so the student is of necessity playing the role of the expert.  This is a good role for a student to play in an assessment, since it empowers her to put on her best performance.  If, as in most classroom situations, not to mention most assessments, the student is not the expert, then she is put in a position of wanting to say what the teacher thinks is right, and it's hard for either the student or her teacher to get a sense of the quality of her own independent thought.

When I have a conversation like this, I think my job is to ask the student questions that will get at the quality of her reading and make her thinking more visible to herself and to me--so that she can see what she knows, how she knows it, and what she doesn't know.  I ask different questions depending on the student.  Sometimes I start out with a general question.  If the student is just starting a book, I might ask, "Are you liking it so far?" Whether the answer is yes or no, I might ask, "Why?"  Or: "What's good about it?" Sometimes, instead of starting with a question, I start by having the student read to me for a half a page or so.  This is another kind of assessment, and one I don't even have to give feedback on--the kid can hear, herself, where she is stumbling.

When students have to stop and think
 As I follow up, either on my original question or on the passage the kid read to me, the questions narrow to a particular focus, and very quickly we get to a question that the student doesn't know the answer to already.  In some cases these questions are ones that I would expect to be obvious, like, "Why is the main character so angry at her friend?" In that case, the student's confusion is very interesting, since we would seem to be identifying a basic problem with comprehension, yet doing so in a way that might seem inherently interesting, and in a way that encourages the student to find out the answer, rather than, as on a standardized assessment, putting the emphasis on the student's failure to figure out what the teacher or state already know, and with little opportunity for immediate follow-through.

In other cases the questions that give the student pause are more difficult or literary, like "Why do you think the author started the book with this scene?" or "How could you tell that she was angry?"  In these cases, too, we are noticing what the student has already considered and what she has not given a thought to.  (At other times, the students get something obviously wrong, and the teacher can follow up in a gentle and friendly way and allow the student to figure out for herself what she was confused about and why.)

One thing that's striking in doing these one-on-one conversations is how quickly we get to points at which the students need to stop and think before they respond.  This stopping and thinking is a pretty big difference between thoughtful intellectual conversations and the usual adolescent repartee. As I remember from my own youth and observe in my students, adolescent conversations are mainly about loud, immediate disagreement or loud, immediate agreement. The loudness and immediacy overwhelms most of the potential for thoughtful critical analysis.  One-on-one conversations in the classroom, conducted in a whisper and aiming less at feel-good agreement or dramatic disagreement, are dramatically different, not least because they lead to so much stopping and thinking.

My guess is that this stopping and thinking is when much of the learning happens, as students see what they understand and what they don't, and as they think through new ideas that they haven't thought about before.  For the conversation is not only assessment, but is also a form of instruction.  In my questioning I am instructing them in ways of looking at a book, in categories of literary thought, in literary vocabulary, and so on.

Disadvantages of this method of assessment
The major disadvantage of these individual conversations is that each student can't get very much of my time.  If it takes a couple of minutes for the class as a whole to settle down enough for me to start talking to kids individually, and if each conversation takes four minutes, and if also I want to quickly check on what progress the other kids in the class have made, then I can get through four conversations in a twenty-minute independent reading period.  That means, for my sixteen-student classes, that I can talk to each kid individually for four minutes each week.  That's not very efficient.

Another disadvantage is that the assessment is not uniform.  I'm not checking each kid against the same benchmark, so it's not easy to compare.  Another disadvantage is that these assessments are often random, coming organically out of whatever passage the kid happens to be reading right then.  Of course, these two disadvantages are also advantages, since the lack of uniformity means that the assessments are better suited to the individual students and the randomness of the passages often sparks my thinking in ways that I couldn't have anticipated.

A last disadvantage is that it's been hard, at least for me, to keep good records of this kind of qualitative, individualized assessment, so it's hard to measure progress and to follow up.  I have to confess that in my preliminary experiments with this kind of assessment, I haven't yet figured out a good record keeping system.  It needs to be very simple, because I, like Ben Franklin, am not very organized.  I'm going to work on this over the next month or so, and I'll follow up with another post, in which I also give some more specific examples of these kinds of conversations.

Most people think about assessment in the same way Mr. Google does (google "reading assessment" to see what I mean): that is, as standardized tests, usually written, administered, by all-knowing adult authorities, upon children who are probably all too aware that (1) they're being tested and (2) that the assessment is of very little interest to either the adult or kid except as an assessment. So perhaps the best thing about using an informal conversation about an independent reading book as an assessment is that it doesn't feel like an assessment.  All of what I'm saying here seems incredibly obvious--probably even when Rousseau said it it seemed pretty obvious--but it might be worth reminding ourselves that assessment is about more than just testing.

Post Script: Similarity to what happens naturally in a literate family; limitations of school
The kind of conversations I've discussed are essentially like the conversations that we have with children in our own homes, starting with the conversations we had when we were reading picture books to them.  The fact that I think doing this in a classroom for four minutes a week is worthwhile is quite amazing, given that many four-year-olds get this kind of treatment for twenty minutes every single night.

This points, perhaps, to the limitations of school.  There's no way that I can possibly do as good a job, as an English teacher responsible for the reading and writing of 85 children, as I do as a father responsible for the reading and writing of two children.  In a sense that's okay--as long as what they do with me is worthwhile--but it's worth remembering the limitations of the system in which we work.

Friday, February 1, 2013

My colleagues and I gave a common assessment!

Leafstrewn is renowned for its independent and distinctive teachers.  Or, some might say, we're notorious for our lack of consistency.  In any case, every teacher has her own way of doing things.  We can’t agree on common policies and practices, let alone enforce them. If ten people discuss an issue, there will be passionate defenses of twenty different opinions.  Our English department chair uses the phrase “herding cats” more frequently than she probably realizes.  So when the ninth grade teachers—all ten of us—decided to make half of our midyear exam a common assessment, I wasn’t sure if it would actually happen.

Amazingly enough, we did it.  Every freshman at Leafstrewn was given the same two-page passage from a Sherman Alexie story and the same prompt for an analytical paragraph, and every paragraph was graded with the same 5-category rubric.  To make the grading more objective, we graded the exams, not of our own students, but of our colleagues’.

We will eventually sit down and consider the numbers, but I have a few reactions now:

(1) reading a passage and writing an analysis of it is an extraordinarily complex task, which is great, but it’s pretty hard to assess in an objective way;
(2) the strength of rubrics is that they are specific and explicit, but this is also their weakness;
(3) everyone got a B;
(4) that’s okay!
(5) I could have done a much better job of preparing my students, and that preparation would have been better not just for this assessment, but in general.

(1) Reading a passage and writing an analysis of it is an extraordinarily complex task, which is great, but it’s pretty hard to assess in an objective way

There is an incredible amount to keep track of when you’re reading anything closely—emotions, connotations of particular words, figures of speech, intertextuality, patterns (like repetition and contrast) within the text, etc.  Writing, too, is really, really complicated—you have to master grammar, ideas, structure, logical arguments, relevant evidence, and so on.  Skilled readers and writers do all this unconsciously, and we sometimes forget that it is amazingly complicated, and our brains, even the most “limited” of them, are quite incredible.

This incredible  complexity becomes much more visible when you start talking about how to judge the quality of student reading and writing.  Different teachers have different ideas of which pieces of this incredible complexity to focus on.  It’s like the story of the elephant: I’m looking at the elephant’s legs, one colleague is looking at the trunk, another is bumping into the tusks, and so on.  Designing a rubric is tough, because there are always things that you’re leaving out, or looking at from only one side.

(2) The strength of rubrics is that they are specific and explicit, but this is also their weakness

Much of the trouble in grading the assessment came in using the rubric.  A rubric is intended to make the grading more transparent and clear, and, most crucially, more specific.  If a student is told, “Your essay is bad,” the student will want to know what in particular was bad about it.  A rubric is supposed to offer the kid that specificity.  What was interesting about the grading process was that the specificity of the rubric was usually exactly what caused trouble in the grading.

For example, if we judged that a conclusion is not good enough for the “Good” category, we had to circle the box on the rubric for a conclusion that “Needs Work.” That box reads: “Brings paragraph to a finish that repeats previous ideas.”  The problem is that there are many ways for a conclusion to be bad, and repeating previous ideas is only one of them. I ran into this problem of overly specific descriptors in every single category on this rubric.

(3) Everyone got a B
Either because our rubric was too easy at the low end, too tough at the high end, or because our students are all pretty good, or because we did a good job of preparing our students, most of the grades fell in the B range. 

(4) That’s okay!
I think one of the lessons here is that actually for all our hand-wringing, our kids are really quite competent.  They can read a passage and write a reasonable paragraph about it.  They are not illiterate.  Almost all of them managed to come up with identifiable topic sentences, evidence that more or less supported their main ideas, and a conclusion that in some way related to what they were saying.  This is no mean feat for a fourteen year old, and I wonder, I admit, if it has something to do with MCAS.  Maybe, as our department chair says, MCAS has really improved kids’ ability to write these kinds of paragraphs.

(5) I could have done a much better job of preparing my students, and that preparation would have been better not just for this assessment, but in general!

I think this common assessment was a great thing to do.  Having students read something and analyze it in a disciplined way is worthwhile, and doing it as a group certainly made my own teaching better.  I was more focused, my students were more motivated, and it took some of the dissonance out of the grading.  (Normally, when we grade our own students’ work, there is an uncomfortable dissonance.  It is as if Bela Karolyi were to judge his own gymnasts’ routines, or as if a soccer coach were training his team for a game against herself.)  In a fairly short and stress-free preparation, I think I did a reasonably good job.

Nevertheless, although my students' performance was fine, there was a lot of room for improvement. How could I have helped them more?  What could I have done better?  A bunch of things, but here's one: I didn’t train my students well enough in coming up with a good main idea.  They tended to say something like, “The impression the author creates in this impression is of a family that is struggling.”  That is pretty obvious, and I need to help my students learn how to go deeper. To take an obvious thought and push it deeper one may:

  * Explore the why of the obvious thought (e.g. the family struggles because they’re in denial).

  * Consider ways in which the opposite is true as well, and craft a semi-dialectical topic sentence of the "Although A, nevertheless B" type--and then by the end of their paper they may arrive at the synthesis of C.  Later in life they can worry, Mr. Ramsay-like, about getting to Q or Z.

  * Explore the how of the obvious thought (e.g. the family struggles ineffectually, trying the same things over and over again even though they produce no results (father looking in wallet over and over, son dreaming over and over, etc.))

* Are there other ways?  Applying a schema? Making a connection?  What else? 

Teaching my students to push their thinking further would be useful not merely for the exam, but in general.  This is something that would be useful to focus on explicitly, and that I somehow overlooked.  That is one of the virtues of this common assessment--it makes the whole process more conscious and transparent, and so lets us see things that we should have seen before.

In the end, though, we and our students did a fine job.  Now if we could just get them to like to read...