Fool's Gold from Hoxby

If her study is above reproach,

Why hide the data?

In September, conservative Hoover Institution economist Caroline Hoxby released an update to her 2007 study of charter schools in New York City. Hoxby claimed that compounded gains for an average student continuously enrolled in third to eighth grade in a charter nearly closed the “Harlem-to-Scarsdale” achievement gap and implied that the trend would continue, although that was apparently outside of the scope of her data. Technical report here.

At last, charter school advocates had some good news in a sea of equivocal data. The news was so good that pro-charter political groups began broadcasting Hoxby-as-gospel without pausing for that customary scholarly requirement; a peer review.

Why should they? The Hoover Institution's mission is "collecting knowledge, generating ideas, and disseminating both" in support of "private enterprise"as opposed to "governmental, social or economic action." According to Jim Horn at Schools Matter, it was charter advocates who ordered the study - which one assumes, is what rich politicians do when they don't like the existing studies.

It was a huge embarrassment to Gates, Broad, Bloomberg, and other corporate school charterizers when the first peer-reviewed national study of charter schools showed that charters outperform public schools in less than 20 percent of the cases. Since then the oligarchs have brought in the hired gun economist, Catherine Hoxby, who never saw a charter school she didn't like, to slime the study as best she can.

Bloomberg has even sweetened Hoxby's pot, asking her for a good-news study of New York's charter schools, a study whose methodology Hoxby immodestly refers to as reflecting the "the gold standard."

Princeton's Jesse Rothstein objected to Hoxby's method in an earlier study in a series of back and forth papers where he complains that,

1) Hoxby has not made available the data that were used for her original paper.
2) Hoxby's basic result holds only when we rely on her specific construction of the larger streams variable. She has never presented estimates that do not rely on this variable; all such estimates that I have computed yield small, insignificant effects of choice on test scores.
3) There are several odd aspects of Hoxby.s particular larger streams variable that cast doubt on its validity.
4) There are serious errors in both the program code and data that Hoxby has distributed.
5) Hoxby has not released the code that was used in her Reply, but she offers no indication that these errors have been repaired.

Once Rothstein published that he was unable to replicate Hoxby's study due to its flaws, she was quick to suggest in an interview with the Harvard Crimson, that the dispute was part of a larger issue of “race and gender bias” in higher education. Hoxby is black, while both Rothstein and his dissertation adviser at University of California, Berkeley, Professor David Card, are white.

Hoxby was similarly skewered over the 2009 study by the late Arizona State University research debunker Gerald Bracey, in the Huffington Post. Bracey wrote,

The study was conducted by economist Caroline Hoxby, the only person in the whole country who consistently finds results that favor charters. Here are a few cogent items about the study:

1. Many of the data that would be needed to draw conclusions are not presented.
2. The study is limited to New York City.
3. The study has not been peer reviewed.
4. The study was published by a pro-charter advocacy group staffed with people
   who used to work in charter schools.
5. The editorial writer, Jo-Ann Armao, lacks the background in econometric research to actually know how to interpret the study. She was, therefore, engaging in faith-based editorializing but passing it off as evidence-based.
6. Even if the study proves to be sound (unlikely according to some other researchers who have also looked at it), it is only ONE study. Strong conclusions in any field should never be drawn on the basis of only one study.

Where did all this furor begin? Over at SusanOhianian.org, Bracey tells the story of how certain well-funded private think tanks ignored the evidence (because it disagreed with their ideology) and moved the political playing field from the laboratory to the newspapers.

The "gold standard" Hoxby likes to refer to is the mere existence of a lottery used at most charter schools when applicants outnumber capacity; known as being oversubscribed. But this would never pass for gold standard research in biomedical research.

Hoxby's study is no clinical trial.

The "gold standard" applies to double-blind experimental studies with random assignment. That means that some research participants get the experimental treatment and some get a placebo, and both are assigned randomly. It also means that neither the researchers nor the participants know who is getting which treatment. After all, expectations are important, and the mind can set us up for all kinds of things. The fact that Hoxby's study fails to meet the rigor of the gold standard seems to bother her fans - not at all.

Where is the placebo?

Without a placebo subjects who know they are receiving a treatment may behave differently. On the other hand, what impact might non-selection have on a student denied entry to their desired school?

What about peer effects?

From a personal level, if you have a child, you don't care about controlling for peer effects. Actually, you want the effects to be left in so that you can take advantage of them. If charter schools have "better" students, that's a reason to send your own child to a charter school. However, if this analysis is done for policy purposes, to influence policy-makers, then peer effects do matter. If you are thinking about all students, not just the select few who can get into the "better" school, you need to control for peer effects.

Then there's selection bias on the School Level .

The goal of this lottery-based study design is to avoid self-selection bias in the data. However, those who use it do not acknowledge the additional selection problems they create.The most important problem is that not all charter schools are oversubscribed, so not all charter schools can be included in these studies. This wouldn't be a problem if we had good reason to believe that a random selection of charter schools were included, but that is obviously not the case. Clearly, the "better" charter schools are far, far, far more likely to be oversubscribed than the "worse" charter schools. This biases the sample rather severely towards better charter schools.

And a really strong public school is not going to lose a lot of students to a simply above-average charter school; which introduces another bias toward charters.

Finally generalizability remains a problem.

Lottery-based studies only include the kinds of students and families that apply to charter schools in the first place. Even if the previous issues could be corrected, how can one know that other sorts of students and families would see the same benefits? The fact is that different populations might benefit less or more from going to a charter school. It is simply impossible to know from this kind of study. Of course, if you are only concerned about benefitting the kids of families who already opt for charter schools, then this is not a problem. But if you aim to help a broader population than that, you need a better methodology.

As Bracey put it,

In June, 2009, another study of charters--this one more of a national look-- concluded that 46% of charters did no better than comparable public schools, 17% outperformed the publics and 37% did worse. "We've got two bad charter schools for every good one," said Margaret Raymond, the Stanford University researcher who conducted the study. Raymond has been known as a charter supporter so her willingness not to flinch in the face of these data is admirable. Would the FDA approve a drug that had adverse effects twice as often as positive effects?

In reality, the rigors of double-blind studies would be nearly impossible to adhere to in social sciences, which, for this and other reasons, should never be compared to biomedical research. Social scientists have relatively poor control over their variables; but sometimes like to act as if they do.

On a policy level, we need to be concerned with charters more generally than that. If we ... approve new charter schools, we have to expect an average charter school to result, not an exceptional one.

Hoxby compares test results of students enrolled in NY’s charter schools to the results for those students who applied to a charter but were not selected for admission. This approach shines light on the probability that charters enroll more motivated families.

As the former principal of Cassidy School, I can attest, first-hand, to the power of a unified team of adults (parents and teachers with resources) working cooperatively for the benefit of all of their children. I suspect it is this "motivation" that is the cause of stronger student achievement. To the extend that those motivational forces can be marshaled in any given public, or charter, or private school; I think you get a much better school. I suspect the "charter school effect" is some manifestation of that teamwork by a community of responsible, hard-working adults.

This from EdWize:

Hoxby’s findings are encouraging: by the third grade, the average charter school student was 5.8 points ahead of the lotteried-out counterpart in math and was 5.3 points ahead in English Language Arts. As Hoxby follows students’ achievement from 2001 to 2008, she also finds that the average charter school student gained 3.6 more points each year in math and 2.4 more points each year in ELA. For an average charter student continuously enrolled in grades four through eight, the effect is larger with annual gains of 5.0 points in math and 3.6 in ELA above the performance of the lotteried-out student. (Last year, nine charters enrolled students across all of these grades.)

Such a dramatically-presented conclusion is sure to feature prominently in charter advocates’ efforts to expand the number of charter schools across the city and state. And if it’s true, then why shouldn’t we?

Even if we assume Hoxby's data are correct - a rather large assumption under the circumstances - states ought to be somewhat concerned about resegregation.

Hoxby finds that 92 percent of charter students are black or Hispanic, compared to 72 percent in district schools and concludes that “the existence of charter schools in the city therefore leaves the traditional public schools less black, more white, and more Asian.” Such racial segregation is consistent with research on charter schools in other states including North Carolina, Texas and elsewhere.

Although this statistic is likely to be a function of charter schools’ location in largely black and Hispanic neighborhoods, Hoxby also reports that fewer white students are applying to the charters; although 14 percent of residents in the charter school
neighborhoods are white non-Hispanic, only 4 percent are applying.

Additionally, Hoxby confirms that the city’s charters enroll fewer English language learners (4 percent) than district public schools (14 percent) — despite their location in Hispanic communities.

On another key measure of economic diversity, it’s puzzling that Hoxby compares charter school free- and reduced-price lunch participation rates, which she pegs at 91 percent, to the citywide rate of 72 percent. For a report that prides itself on apples-to-apples comparisons, this charter-to-city comparison is a glaring inconsistency. As Edwize has demonstrated here and here, a better comparison is between charters and the surrounding district/feeder schools. By this comparison, charters have fewer students eligible for free lunch, suggesting that the city’s charters are more affluent than the neighboring district schools.

It’s also worth remembering that charter schools do not have attendance zones and can, in theory, draw students from across neighborhoods. But as the demographic statistics reveal, if charters are attracting applicants from beyond their immediate neighborhoods, it is not generating more racial, ethnic, and economic diversity.

Incidentally, Hoxby’s updated report makes no mention of student attrition from charter schools to district schools, despite the fact that she has access to student-level data over many years.

Presumably she can, and should, confirm or reject the anecdotal reports of this occurrence, as student attrition biases achievement results. Attrition is also a measure of equity, as district schools must educate all of the students who come through their doors while charters, with site-specific discipline codes and the power to expel students, do not.

Moreover, the report provides tacit evidence that all of the students and families who apply for a charter are in fact different from the other public school parents and students who don’t apply. Whether they are more motivated, place a higher emphasis on education, or some other unmeasurable characteristic, Hoxby finds that “lotteried-out students’ performance does improve and is better than the norm in the U.S. where, as a rule, disadvantaged students fall further behind as they age.”

Given this and the other demographic findings, it is unclear how the report “lays to rest the persistent myth that charter schools “cream” the best students,” as one charter advocate maintains. Quite the opposite — the achievement results for all charter applicants suggests there is likely to be some truth to the creaming “myth.”

Hoxby's data ought to give pause to those charter advocates who are concerned with issues of social justice. Any system that resegregates school children by race is suspect. It is not a satisfactory explanation that such resegregation is being caused by parental choice; segregation has always been caused by parental choice. It is not conclusive that charters have that effect, but Hoxby went some distance showing that it just might.

Hoxby maintains that her method is the “gold-standard” approach when seeking to determine the unique impact of charter school effects, but EdWize questions "two complications that would otherwise moderate her findings."

First, it is conceivable that low-performing charters are undersubscribed and may not have a large “lotteried-out” population to study. In effect, this could under-weight the results of poor performing charters and biases her data in favor of higher-performing charters. A technical version of the paper or scholarly access to her data could ascertain if this is in fact occurring. And despite the reports sweeping conclusions, there are low-performing charters in her study: Hoxby finds that 31 percent of students attend a charter that is performing no better than or worse than the schools attended by lotteried-out counterparts in math and 24 percent are doing no better or worse in ELA.

Second, it’s not clear that Hoxby’s methodology takes into account peer effects — the straightforward concept that students not only learn from their teachers but are also influenced by their peers. Charter schools benefit from the fact that 100 percent of their students hail from motivated families; as a result, a charter student is surrounded by peers who are there by choice — rather than by attendance zone. In contrast, the “lotteried-out” students may not benefit from such an intensive peer effect...

This from Equity Overlooked - Charter Schools and Civil Rights Policy:

The expansion of charter schools is a central policy focus of the Obama administration. Charter schools encompass a variety of schools with different priorities serving many communities and students from a range of backgrounds. There are outstanding and diverse charters, some of which have been highlighted
by this or prior administrations.

While the administration has acknowledged the importance of regulating and closing low performing charter schools, it has yet to respond to concerns raised about continued racial isolation in charter schools. Why is this lack of civil rights oversight so troubling? Without necessary safeguards against the segregating effects of charter schools, disadvantaged families are left to comprehend and cope individually with the complicated landscape of school choice.

Access to the educational marketplace is unequally constrained by a number of factors, including contact with advantaged social networks through which information regarding school quality is exchanged, language barriers, socioeconomic status and the ability of parents to arrange transportation for their schoolchildren. Unless proactive equity measures – like extensive outreach and free transportation – are embedded in the design of charters, and subsequently monitored and enforced, this popular version of education reform simply reinforces unequal educational opportunity.

In the 61-page "Charter Schools' Performance and Accountability: A Disconnect," Bracey summarized the research of others, not doing any original work--found a number of studies showing poor charter school performance.

Diane Ravitch, who served as the Education Department’s assistant secretary for educational research and improvement under President George H.W. Bush, doubts the evidence upon which two of the Obama administration's education efforts are based.

“What is extraordinary about these regulations is that they have no credible basis in research. They just happen to be the programs and approaches favored by the people in power,” writes Diane Ravitch, a research professor of education at New York University, in her blog, Bridging Differences, which is hosted by edweek.org.

In the meantime, a new 2009 study of Baltimore charter schools concluded:

• The number of charter schools increased from 12 in 2005-06 to 22 in 2007-08
• Enrollment in charter schools increased from 2,925 students in 2005-06 to 5,520 in 2007-08, an increase of almost 89%.
• There is significant variation among charter schools on most measures of school and student performance.
• On average, charter school students are less disadvantaged than non-charter school students. Charter school students are less likely to be eligible for special education services, to be over-age for their grade, or to be FARMS-eligible.
• Overall, charter schools do not enroll a higher proportion of students from outside the district or from non-public schools than non-charter schools. There is, however, significant variation among individual schools in the enrollment of students new to City Schools, with selected schools enrolling a larger proportion of students from outside the district.
• Some charter schools attract a significant proportion of their enrollment from surrounding non-charter City schools.
• Charter school students are less likely to leave the district at the end of the school year and are more likely to re-enroll in their schools the following school year compared with non-charter school students.

The fact is the evidence on charter schools is not in. Some are great. Some are terrible. Most are somewhere in the middle. Just like public schools.

Charters outperform public non-charters is some places (apparently both Hoxby and CREDO agree that New York is one such place) and the reverse is true elsewhere.

The reason for the performance of students in the best schools (of any variety) seems less likely to be due to anything specific to charters. Rather, it is more likely to be due to site-specific conditions of parental and teacher motivation to build a good school.

The amount of time students spend engaged in academic work also seems to be central to school success. In some cases, under binding arbitration, union restrictions on the amount of time a teacher may be asked to work, might play a role.

Political think tanks that promote Hoxby's study as definitive are suspect for political and ideological motivation and provide strong evidence that they are not truly research facilities.

All this being said, Kentucky still has several schools that have languished in tier 4 and 5 purgatory for some time now. We must assume that the school districts where these schools exist, have faithfully tried, but failed to make a difference for the children under their care. In such cases, resisting a fresh new approach - even a privately run approach like charters - suggests a greater devotion to the feelings of adults than the needs of students. Kentucky needs a strong, but limited, charter law that continues the effort to produce a high-quality school in every community.