Hoxby: Too early to draw sweeping conclusions on charters

In the wake of KET's recent discussion of charter schools the communication lines aroud here have perked up.

Some have asked me to look into Sheldon Berman's statements, and I will - next chance I get. It is too soon to draw conclusions about charters. That includes calling charter schools a failed reform. That is nonsense. Just like sclling charter schools a civil right is nonsense.

One KSN&C reader wanted to know why I remain unconvinced by Carolyn Hoxby's research which suggests that New York's charter school students outperform their public school peers. The reader asked why I don't love her method of comparing charter school students with nearby public school students.

Well, in part, it's because I'm not sure they are peers. The other part is an admitted suspicion of psuedoscientific work from some institutions that collect (buy) like-minded researchers and then set out to create data in support of their predetermined worldview.

I have no idea how to guard against this - except to remain suspicious, think critically, and to wait for scholarly confirmation through peer review before jumping to any conclusions.

Speaking of which, there's this from "Advantage None: Re-Examining Hoxby’s Finding of Charter School Benefits" by Lawrence Mishel and Joydeep Roy on Hoxby's 2004 study:

Hoxby's analysis, however, suffers from the fact that her method of comparing charter schools to their neighboring regular public schools (and to those neighboring public schools with a similar racial composition) inadequately controls for student backgrounds. In her sample of matched schools there are often significant differences in the demographic and socioeconomic characteristics of the students. For instance, comparing the charter schools in Hoxby's sample to the matched neighboring public schools with a similar racial composition shows that the charter schools have a disproportionately higher black population (34% vs. 28%) and higher white population (43% vs. 36%), while the share of Hispanics is lower (18% vs. 30%). Her sample of charter schools also has disproportionately fewer low-income students than does the matched "racially similar" sample of neighboring public schools (49% vs. 60%). The same picture emerges in terms of the demographics of charter schools in central cities: charter schools serve a disproportionately lower share of minorities and low-income students compared to their matched regular public schools. Thus,
without further controls, Hoxby's method of comparing "racially matched" schools does not appear to be effective in controlling for student characteristics.

Hoxby's result of a positive charter effect on math proficiency disappears when racial composition is controlled for directly. Further, when both racial composition and low-income status are controlled for, the positive effect of attending a charter school disappears for both math and reading (it becomes very small and not statistically significantly different than zero). Thus, Hoxby's conclusion that "although it is too early to draw sweeping conclusions, the initial indications are that the average student attending a charter school has higher achievement than he or she otherwise would" (Hoxby 2004a) does not hold up against direct controls for student background.

Plain and simple - the jury is still out on the performance of charter school students relative to public school students.

Now for those who want me to quit doubting Hoxby and start lovin' her findings, particularly in the wake of CREDO's recent softening, my answer is - I can't. At least not yet. Let's have some pros give her data the twice over before we start promising the public something we cannot deliver.

In the meantime, surely skeptics and hopeful romantics alike can agree with Hoxby when she says it's "too early to draw sweeping conclusions."