Thursday, December 20, 2012

Can we trust educational research? ("Visible Learning": Problems with the evidence)

I've been reading several books about education, trying to figure out what education research can tell me about how to teach high school English. I was initially impressed by the thoroughness and thoughtfulness of John Hattie's book, Visible Learning, and I can understand why the view of Hattie and others has been so influential in recent years.  That said, I'm not ready to say, as Hattie does, that we must make all learning visible, and in particular that "practice at reading" is "minimally" associated with reading gains.  I discussed a couple of conceptual issues I have with Hattie's take in an earlier post--I worry that Visible Learning might be too short-term, too simplistic, and less well-suited to English than to other disciplines.  Those arguments, however, are not aimed at Hattie's apparent strength, which is the sweep and heft of his empirical data.  Today, then, I want to address a couple of the statistical weaknesses in Hattie's work.  These weaknesses, and the fact that they seem to have been largely unnoticed by the many educational researchers around the world who have read Hattie's book, only strengthen my doubts about the trustworthiness of educational research.  I agree with Hattie that education is an unscientific field, perhaps analogous to what medicine was like a hundred and fifty years ago, but while Hattie blames this on teachers, whom he characterizes as "the devil in this story" because we ignore the great scientific work of people like him, I would ask him to look in the mirror first.  Visible Learning is just not good science.

Hattie's data
Visible Learning attempts to be both encyclopedia and synthesis. The book categorizes and describes over 800 meta-analyses of educational research (altogether, those 800 meta-analyses included over 50,000 separate studies), and it puts the results of those meta-analyses onto a single scale, so that we can compare the effectiveness of the very different approaches.  After categorizing the meta-analyses, into, for instance, "Vocabulary Programs", "Exposure to Reading", "Outdoor Programs", or "Use of Calculator", Hattie then determines the average effect that the constituent meta-analyses show for that educational approach.  By these measures, exposure to reading seems to make more of a difference than the use of calculators, but less of a difference than outdoor programs, and much less of a difference than vocabulary programs. (There are some odd results: "Direct Instruction," according to Hattie's rank-ordering, makes more of a difference than "Socioeconomic Status.")

Like other teaching gurus and meta-meta-analyzers (for instance, Robert Marzano, whose 2000 monograph, A New Era of School Reform, makes the case very explicitly), Hattie believes that good teaching can be codified and taught (that sounds partly true to me), that good teaching involves having very clear and specific learning objectives (I'm somewhat doubtful about that), and that good teaching can overcome, at the school level, the effects of poverty and inequality (I don't believe that).  Hattie uses a fair amount of data to back up his argument, but the data and his use of it are somewhat problematic.

First, questions about the statistical competence of Hattie in particular
I am not sure whether we can trust education research, and I am not alone.  John Hattie seems to be a leading figure in the field, and while he seems to be a decent fellow, and while most of his recommendations seem somewhat reasonable, his magnum opus, Visible Learning, has such significant issues that my one friend who's a professional statistician believes, after reading my copy of the book, that Hattie is incompetent.

The most blatant errors in Hattie's book have to do with something called "CLE" (Common Language Effect size), which is the probability that a random kid in a "treatment group" will outperform a random kid in a control group.  The CLEs in Hattie's book are wrong pretty much throughout.  He seems to have written a computer program to calculate them, and the computer program was poorly written.  This might be understandable (all programming has bugs), and it might not have meant that Hattie was statistically incompetent, except that the CLEs Hattie cites are dramatically wrong.  For instance, the CLE for homework, which Hattie uses prominently (page 9) as an example to explain what CLE means,  is given as .21.  This would imply that it was much more likely that a student who did not have homework would do well than a student who did have homework.  This is ridiculous, and Hattie should have noticed it.  But even more egregious is when Hattie proposes CLEs that are less than 0.  Hattie has defined the CLE as a probability.  A probability cannot be less than 0.  There cannot be a less than zero chance of something happening (except perhaps in the language of hyperbolic seventh graders.)

As my statistician friend wrote me in an email, "People who think probabilities can be negative shouldn't write books about statistics."

Second, doubts about the trustworthiness of educational researchers in general
My statistician friend is not the first to have noticed the probabilities of less than zero. A year and a half ago a Norwegian researcher wrote an article called "Can We Trust The Use of Statistics in Educational Research" in which he raised questions about Hattie's statistical competence, and in follow-up correspondence with Hattie the Norwegian was not reassured.  (Hattie seems, understandably, not to want  to admit that his errors were anything more than minor technical details.  In a exchange of comments on an earlier post on this blog, as well, Hattie seems to ignore the CLE/negative probability problem.)

For me, the really interesting thing about Hattie's exchange with the Norwegians was that he seemed genuinely surprised, two years after his book had come out, by the fact that his calculations of CLE were wrong.  In his correspondence with the Norwegians, Hattie wrote, "Thanks for Arne Kåre Topphol for noting this error and it will be corrected in any update of Visible Learning."  This seems to imply that Hattie hadn't realized that there was any error in his calculations of CLE until it was pointed out by the Norwegians--which means, if I'm right, that no one in the world of education research noticed the CLE errors in between 2009 and 2011.

If it is true that the most prominent book on education to use statistical analysis (when I google "book meta-analysis education", Hattie's book is the first three results) was in print for two years, and not a single education researcher looked at it closely enough and had enough basic statistical sense to notice that a prominent example on page 9 of the book didn't make sense, or that the book was apparently proposing negative probabilities, then education research is in a sorry state.  Hattie suggests that the "devil" in education is the "average" teacher, who has "no idea of the damage he or she is doing," and Hattie approvingly quotes someone who calls teaching "an immature profession, one that lacks a solid scientific base and has less respect for evidence than for opinions and ideology" (258).  He essentially blames teachers for the fact that teaching is not more evidence-based, implying that if we hidebound practitioners would only do what the data-gurus like him suggest, then schools could educate all students to a very high standard.  There is no doubt that there is room for improvement in the practice of many teachers, as there is in the practice of just about everyone, but it is pretty galling to get preachy advice about science from a guy and a field who can't get their own house in order.

Another potential problem with Hattie's data
Aside from the CLE issue, I am troubled by the way Hattie presents his data.  He uses a "barometer" that is supposed to show how effective is the curricular program or pedagogical practice he is considering.  This is the central graphic tool in Hattie's book, the gauge by which he measures every curricular program, pedagogical practice and administrative shift:




Note that developmental and teacher effects are both above zero. What this implies is that the effect size represented by the arrow is not the effect as compared to a control group of students that got traditional schooling, nor even the effect size as compared to students who got no schooling but simply grew their brains over the course of the study, but the effect size as compared to the same students before the study began.

This would imply that offering homework, with a reported effect size of .29, is actually worse than having students just do normal school, or that multi-grade classes, with an effect size of .04, make kids learn nothing.

Now, that is obviously not what Hattie means.  The truth is that Hattie sometimes uses "effect size" to mean "as compared to a control group" and other times uses it to mean "as compared to the same students before the study started." He seems comfortable with this ambiguity, but I am not.  Not only is the "barometer" very confusing in cases like homework and multi-grade classrooms, where the graphic seems clearly to imply that those practices are less effective than just doing the regular thing (especially confusing in the case of homework, which is the regular thing), this confusion makes me very, very skeptical of the way Hattie compares these different effect sizes.  The comparison of these "effect sizes" is absolutely central to the book.  Comparing effect sizes (and he rank orders them in an appendix) is just not acceptable if the effects are being measured against dramatically different comparison groups.

Hattie, in a comment on an earlier post in which I expressed annoyance at this confusion, suggested that we should think of effect sizes as "yardsticks"--but in the same comment he says that effect size is the effect as compared to two different things.  In his words: "An effect size of 0 means that the experimental group didn't learn more than the control group and that neither group learned anything."  Now, I am an English teacher, so I know that words can mean different things in different contexts.  But that is exactly what a yardstick is not supposed to do!

Of course, it is possible that many of Hattie's conclusions are correct.  Some of them (like the idea that if you explicitly teach something and have kids practice it under your close observation, then they will get better at it more quickly than if you just ask them to try it out for themselves) are pretty obvious.  But it is very hard to have much confidence in the book as a whole as a "solid scientific base" when it contains so much slipperiness, confusion and error.

Beyond these broad issues with Hattie's work, I also have some deep qualms about the way he handles reading in particular.  Maybe one day I'll address those in another post.

61 comments:

  1. Look, I agree about the moving target, but the bit about 0.29 for homework is, as I read it, saying that you want to consider your resources allocation carefully, and not throw a whole lot of effort into something that is a *relatively* "low performer" **without carefully considering the input**. He goes over and over that this takes us to the point of asking questions, and considering the cohort in question in fine detail. The school uniform thing is a great example; it allows us to see that if one was to pursue the uniform on academic achievement (and a few other) grounds, it would be hard to justify, but it does NOT mean that there aren't other grounds upon which any given group might want to employ a uniform in their particular context. And you say about explicit instruction being "obvious"; dude, I can tell you that there are plenty of folks still plugging along with "There's maths in that!"

    ReplyDelete
    Replies
    1. You may be comfortable with Hattie's confident assertions and fuzzy math, but I am not. Obviously we want to consider resource allocation carefully, but Hattie goes on and on about how medieval the teaching profession is, and how we need modern scientists like him to bring us into the modern world; if that is his position, it is simply ridiculous for him to be so vague about what his numbers mean.

      Delete
    2. Nice article on criticism of Hattie. When he reviews thousands of studies, he cannot control how any of them are done, and none of them is done in exactly the same way. This blame about the teachers for the unscientific nature of education. We do agree sometimes the education is enforced without any point.

      Delete
  2. Thanks for a nice introduction to the criticism of Hattie. It is being pushed (and over-simplified) in my school. I have two issues that maybe you can speak to a bit. I have not read the book, I have just seen several presentations that are at least once-removed, if not twice removed. 1. The areas researched are presented in a rank ordering. This sends the message that "feedback" is better than "teacher-student relationships" or that "not labeling students" is more effective than "concept mapping" when these seem totally unrelated. 2. I have seen his effect size reported as r and as CLES. Either way, there shouldn't be any values over 2, right? Am I missing something? Just like 0 is a weird number, even values very close to 1 seem out of place and over 1 seem like a disaster. What is going on with that?

    ReplyDelete
    Replies
    1. I would advise you to put these questions to John himself! He is easily accessibel through his university mailaccount. He will explain that some of the items with a lower ranking are a pre-condition for items higher on the list. He specifically uses the example of teacher-student relationships as a pre-condition for effective feedback.

      Delete
  3. Hi Emily: I'm traveling and don't have Hattie's book with me, but certainly my take is that you are right and Hattie's work is basically a mess. More specifically, yes, Hattie seems to stand by his rank-ordering, and we are I think supposed to understand him to be saying that feedback is EXTREMELY important, even more so than teacher-student relationships. Because I don't understand his math, I don't trust this conclusion. But who knows, it is imaginably true... As for your #2, yes, I think the values should be between 0 and 1. But as I've said, I really don't know what these numbers are supposed to represent, and as far as I can tell Hattie himself doesn't know, either. As I wrote in the post, he actually left a comment on my earlier post saying that the effect sizes mean different things for different studies. As far as I'm concerned, his work is pretty much useless. This is, unfortunately, par for the course in educational research, and it wouldn't bug me much except that he has the incredible nerve to blame us teachers for the unscientific nature of education. I'm sorry to hear that his work is being pushed in your school! --EC

    ReplyDelete
  4. Thanks for a great post.

    We desperately want some certainty and some credibility and hell... some hope from education research. I am totally comfortable with Hattie being problematically flawed as the guiding light. It shows how far we need to go. Mostly it raises the question, if Hattie is flawed how ridiculously flawed are the rest of the education experts who preach (and I use that word advisedly) a whole lot of bunk that makes teaching harder and teachers themselves less respected?

    The issue as I see it is that climbing the hierarchy requires holding your tongue about how ineffective a lot of the fashionable, orthodox beliefs about education are.

    ReplyDelete
  5. I believe that you sometimes are holding Hattie responsible for decisions made by the individual researchers he is summarizing. When he reviews thousands of studies, he cannot control how any of them are done, and none of them is done in exactly the same way. Indeed, each effect size IS calculated differently depending on how each study was conducted.

    ReplyDelete
    Replies
    1. Thanks for your comment. But I don't think I am holding Hattie responsible for decisions made by individual researchers. I am objecting to the way he attempts to synthesize and analyze the individual studies en masse. That each individual effect size is calculated differently is exactly my point; each individual effect size is calculated independently, but Hattie's discussion and his "barometer" serve to obscure and confuse these differences, and I am not alone in finding his statistical competence to be suspect. I am no statistician, and statistical incompetence is not a great sin, but Hattie's annoys me because of the pompous and self-righteous way he writes. Again, as I wrote in the post, "He essentially blames teachers for the fact that teaching is not more evidence-based, implying that if we hidebound practitioners would only do what the data-gurus like him suggest, then schools could educate all students to a very high standard. There is no doubt that there is room for improvement in the practice of many teachers, as there is in the practice of just about everyone, but it is pretty galling to get preachy advice about science from a guy and a field who can't get their own house in order."

      Delete
  6. Having spent the day at the Impact Conference in London on Monday and spoken with John Hattie I find this blog intriguing. We have a teacher of English treating his work at face-value rather than analysing it in an exploratory way he might approach a piece of literature, a school where the message has been so reduced that the point of the study has been subverted and RP Phelps clarity of thinking around research analysis, because that is what Hattie's work is.

    The message of the man is surely relatively clear - Visible Learning happens when; teachers plan through seeing learning through the eyes of the student and when students learn through see themselves as their own teachers'. ie knowing what to do when they don't know what to do. The cumulative study highlights what may well be ways to achieve this and it's up to we professionals to select the most effective in ways that suit our situation.

    Effective whole school development, Hattie proports, is where teachers are genuinely supported to collaborate, devise explicit success cirtieria together and build trust across the piece so that teachers get maximum amounts of effective, supportive, developmental feedback and there's a balance of both surface and deep learning for all students.

    The effect size diagram devised by Osiris Educational helps make sense of the range of aspects of learning that we select from. It's not a polemic but a useful graphic. Like Blooms taxonomy, its isn't a linear heirarchy but a shopping list of opportunity to be selected with care to maximise the learning for all.

    ReplyDelete
    Replies
    1. If you found Hattie and his work to be useful, great. But I would warn against thinking that there's much scientific value to it.

      Delete
    2. Gavin it seems an odd suggestion to read a book of meta-analysis as a piece of literature. Unless you mean it's fiction?

      Delete
    3. Gavin it seems an odd suggestion to read a book of meta-analysis as a piece of literature. Unless you mean it's fiction?

      Delete
    4. Gavin it seems an odd suggestion to read a book of meta-analysis as a piece of literature. Unless you mean it's fiction?

      Delete
    5. it definitely is a hierarchy an is used as such - "The major message is that we need a barometer of what works best.." Hattie - in his preface. His appendix then RANKS all the effect sizes from 1- 138. The Australian Government has used his research as a definite hierarchy by cancelling funding to poorer schools based on class size has a low impact.

      Delete
  7. "If it is true that the most prominent book on education to use statistical analysis (when I google "book meta-analysis education", Hattie's book is the first three results) was in print for two years, and not a single education researcher looked at it closely enough and had enough basic statistical sense to notice that a prominent example on page 9 of the book didn't make sense, or that the book was apparently proposing negative probabilities, then education research is in a sorry state".

    I appreciate your perspective, but it's only partially accurate. I have spoken with several quantitative educational researchers, all of who quite frankly do not think about Hattie and certainly don't think enough of his work to read his book. Look at sources like Ed Psych and Measurement, Journal of Ed Measurement, Journal of Educational and Behavioral Statistics, etc. and you will see that there is plenty of complex analysis and statistics being done in the field. The audience for Hattie's book is not serious quantitative educational researchers and we cannot prevent a book like this with its errors from being published and read. Someone at Routledge did a poor job in selecting reviewers, presuming it was subjected to some kind of peer-review initially.

    ReplyDelete
  8. Your qualitative research critique on education very good. Really you finding source too good thanks for share it.

    ReplyDelete
  9. but dismisses their views along these lines as being no more than fragmentary. learn japanese in hong kong

    ReplyDelete
  10. Get some articles about waldorf education to understand better how does it work in general.

    ReplyDelete
  11. The depth of articles can easily be felt of this blog. Very precise and straight to the mark. I understood easily the matter of fact which the author of this blog wanted to deliver through his thoughts. Looking for more.เรียน a level ที่ไหนดี

    ReplyDelete
  12. I moved to England to study and it was very difficult for me to adapt. It's hard for me to write an article or something like this myself, so I mainly use the essays service. I am interested in different language training, so this information is useful to me.

    ReplyDelete
  13. This blog is really helpful for my database. It enhanced the area of my thoughts and pushed me beyond the boundaries. Work ethic of every point is different and represent a new way to improve myself.
    เรียน sat

    ReplyDelete
  14. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts.นิทาน สำหรับ เด็ก

    ReplyDelete
  15. Story of this blog is well written. The writer kept in consideration the grammar very well. Level of English also very well. Lot many new words has been used while writing content of this blog.รับสอนพิเศษที่บ้าน

    ReplyDelete
  16. I found this blog after a long time which is really helpful to let understand different approaches. I am going to adopt these new point to my career and thankful for this help.ติว ged

    ReplyDelete
  17. Thank you very much for writing such an interesting article on this topic. This has really made me think and I hope to read more ---.
    dissertation writing services uk

    ReplyDelete
  18. Graceful written content on this blog is really useful for everyone same as I got to know. Difficult to locate relevant and useful informative blog as I found this one to get more knowledge but this is really a nice one.
    หนังสือนิทาน

    ReplyDelete
  19. Aniket Singh has launched numerous scholarship programmes for underprivileged students at IIT-Madras. Determined to give back to the country and alma mater that has provided him with so much, Aniket has launched the scholarship which helps students cover the costs of tuition and accommodation that they will incur during their four years on campus.
    You can read more about the scholarship at :
    https://www.aninews.in/news/business/business/aniket-singh-launches-scholarship-programme-for-underprivileged-students-at-iit-madras201804181835480003/
    Also visit him at : Aniket Singh
    Check out his book at : Intern Abroad This Summer

    ReplyDelete
  20. Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return.
    Round Bag

    ReplyDelete
  21. I welcome all the suggestion mentioned in this blog related to new learning skills. It is definitely going to help me to adopt new exited way of learning. I think, others will also feel helpful this blog for their needs.เรียนต่ออังกฤษ

    ReplyDelete
  22. As we've seen, meanings of finances made by lenders and political business analysts don't vary significantly. http://www.china-research.net/

    ReplyDelete
  23. Enthusiastic words written in this blog helped me to enhance my skills as well as helped me to know how I can help myself on my own. I am really glad to come at this platform.
    ที่เรียน ged

    ReplyDelete
  24. Mesmerized article written on this blog with other relevant information. It is straight to the point that how we can improve our skills as well as how we can be represented to a new stream of professionalism.อยากเรียนต่อต่างประเทศ

    ReplyDelete
  25. This comment has been removed by the author.

    ReplyDelete
  26. This blog is really helpful to deliver updated educational affairs over internet which is really appraisable. I found one successful example of this truth through this blog. I am going to use such information now.
    ติว ged

    ReplyDelete
  27. Graceful written content on this blog is really useful for everyone same as I got to know. Difficult to locate relevant and useful informative blog as I found this one to get more knowledge but this is really a nice one.
    เรียน ielts

    ReplyDelete
  28. This is really nice to read content of this blog. A is very extensive and vast knowledgeable platform has been given by this blog. I really appreciate this blog to has such kind of educational knowledge.
    excel cursus amsterdam

    ReplyDelete
  29. This blog aware me about different programs which can become very useful for our friends and kids. Few websites provide combined courses and few of the are separately for single subject. Glad to get this information.
    relatietherapie amsterdam

    ReplyDelete
  30. It is really a helpful blog to find some different source to add my knowledge. I came into aware of new professional blog and I am impressed with suggestions of author.
    helderziende amsterdam

    ReplyDelete
  31. In the 2017 updated list of factors influencing student achievement, he still uses negative numbers, so I don't think he considers them "probability;" I think he considers them definite advancing progress. It's the only way negative numbers make sense- these factors actually detract from learning. But if one goes along with that, the list becomes even more problematic and unlikely. It wouldn't matter so much except so many school districts are treating his findings as gospel and putting a lot of eggs in a flimsy basket.

    ReplyDelete
  32. It is really a graceful article and and its application is very much thoughtful step for young minds. We are following these blogs for its very extensive and vast knowledgeable platform.

    ReplyDelete
  33. I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. islamic news in nigeria

    ReplyDelete
  34. our enthusiasm leads you beyond the limits. When you feel yourself enthusiastic that’s the time you can cross any limit. You seek to get perfection by using the ability of work. Read such motivational article and definitely it will help you to know new facts.
    persoonlijke ontwikkeling nederland

    ReplyDelete
  35. หาติวเตอร์ เว็บนี่ดีจริงๆเลยครับ

    ReplyDelete
  36. it's so refreshing to see a post that talks straight to the point. thanks so much for writing about this it has really helped me with building my experience. thanks a lot



    siberian husky puppies for sale near me
    Siberian Huskypuppies
    Siberian Huskypuppies for adoption
      Siberian Huskypuppies breeders near me  

      white Siberian Husky puppies  

    ReplyDelete
  37. its been long since i saw a post that's so educative and informational. i will make sure to share this my facebook group. you can also view contents on our websites below.

    French Bulldog Puppies For Sale

    French Bulldog Breeders

     French Bulldog Puppies For Sale Near Me

    French Bulldog Puppies For adoption

    French Bulldog Puppies


    Blue French Bulldog Puppies

    ReplyDelete
  38. Nice post. I used to be checking constantly this blog and I am impressed! Extremely useful info particularly the ultimate section, I take care of such information a lot.
    CBSE schools in Vellore

    ReplyDelete
  39. köp ketamin
    comprare la ketamina
    買氯胺酮
    ketamint vásárolni
    kupiti ketamin
    Ketamin kaufen
    comprar cetamina
    pirkti ketamino
    osta ketamiini
    osta ketamiini veebis
    pirkti ketamino internete
    comprar cetamina online
    kup ketaminę online
    Ketamin online kaufen
    comprar cetamina online
    купить кетамин онлайн
    kjøp ketamin online
    köp ketamin online
    buy online ketamine
    buy ketamine
    order ketamine online
    ut ketamine online
    beställ ketamin online
    bestill ketamin online
    заказать кетамин онлайн
    Ketamin online bestellen
    online ketamine bestellen
    pedir ketamina en línea




    call/text/<<<<<<<< +1(505)257-5355
    WICK...R>>>>>>ROBERTGREEN1020

    email...bcvsgea1124@gmail.com

    ReplyDelete
  40. Try Ziyyara's online tuition for CBSE Board in Nigeria & get the experienced tutors offering home tuition for CBSE Board in Nigeria.
    Contact no:- +91-9654271931
    Visit On:- tuition classes for CBSE in Nigeria

    ReplyDelete
  41. All thanks to Mr Anderson for helping with my profits and making my fifth withdrawal possible. I'm here to share an amazing life changing opportunity with you. its called Bitcoin / Forex trading options. it is a highly lucrative business which can earn you as much as $2,570 in a week from an initial investment of just $200. I am living proof of this great business opportunity. If anyone is interested in trading on bitcoin or any cryptocurrency and want a successful trade without losing notify Mr Anderson now.Whatsapp: (+447883246472 )
    Email: tdameritrade077@gmail.com

    ReplyDelete
  42. Your blog post is remarkable, its helpful, infact i write about number of topics like market research.. so read and let share insights on content regarding marketing research for business.

    ReplyDelete
  43. Yes, nice topic, the blog is fully detailed and helpful and have good insights you have share over here.. infact i write about number of topics like NRI Lawyer in India let share some views about our content...

    ReplyDelete
  44. Zee interior is a renowned name in the field of Interior designing Patna, they successfully created a niche in the Interior designing industry by providing the best interior designing company in patna영통노블홈타이
    팔달노블홈타이
    노블홈타이
    노블홈타이
    노블홈타이
    노블홈타이
    노블홈타이

    ReplyDelete
  45. Looking for the best online English Classes in Kuwait? Register at Ziyyara to take best online English language classes in Kuwait.
    Contact no:- +91-9654271931, +968-71912179
    Visit On:- english speaking course in Kuwait

    ReplyDelete
  46. Given that we had known about your site a year prior, we'd have been saved the superfluous measures we were taking. Much thanks. David Byrne Big Suit

    ReplyDelete