Fancy math in educational research

I was reading an economics blogger who sometimes writes about inequality, and a post of his (http://owenzidar.wordpress.com/2012/12/29/what-would-happen-if-wed-greatly-expand-charter-school-availability/)sent me to a paper by an economics grad student at MIT about charter schools. The paper (http://economics.mit.edu/files/8429) seemed competent--the author is handier with his math than I am--but basically just marginal embroidery. The paper is another angle on the inadequacy of educational research. Basically, the lesson I take away is that fancy math doesn't matter--what you need is more and better data.  

First, I haven't read that many papers about education that attempt such elaborate mathematical modeling as this one.  Here is a screen shot of one of the pages:

Second, it  was remarkable that such elaborate modeling allowed the author to achieve virtually no new insight. The paper begins by acknowledging that there are two ways to understand the success of Boston charter schools: either the success is due to the non-union faculty and "No Excuses" practices of Boston charters, or the success is due to the self-selected student population. Yes, true--and the author's extremely elaborate model only allows him to conclude that if charter schools make a big difference in the achievement of poorer students, then it will be important to figure out how to get more poor students into charters.

This is an entirely obvious conclusion that you might have reached without doing any of the fancy modeling and without spending years in graduate school at MIT. The key issue here is whether it is the schools or the non-school factors that lead to the higher test scores, and this paper sheds no light on that issue.

Economists often write about education, because of the apparent strengths of both the economists (they're arguably better at math than ed school Ph.D.s) and the education data (it's arguably more abundant and rich than economics data). But this paper shows the limitations of both of these apparent strengths: the economist's fancy math is deployed to no effective purpose; and the education data is clearly inadequate.

If we want to know whether the Boston charter schools are markedly more effective than the rest of the Boston system, we would probably have to conduct an actual scientific experiment: randomly assign a large number of students, not allow the charters to drop their lower-performing students, and see what happened. Until then, I will continue to assume what I have never seen disproven, that schools alone cannot overcome achievement gaps, and that the higher test scores of students in Boston charters does not mean that eliminating teachers' unions and adopting "No Excuses" policies would help poor kids learn more.