Hoxby's Hocus Pocus

A few weeks ago I was discussing Carolyn Hoxby's most recent studies with a fellow blogger and I felt compelled to confess that I couldn't read it. My blogger buddy said he only reads the executive summaries. But that seemed wrong - like asking to be lied to - to me.

Here's why:
Hoxby's regression formula is Greek to me.

So how does one determine whether her methodology is useful? How does one determine if it rises to the level of the "gold standard" as Hoxby claims?

The typical approach is to have professional papers reviewed by knowledgeable peers before issuing the findings in a juried journal. But not these days. When somebody gets a wiff of confirming evidence, straight into the news cycle it goes.

So what's a lowly blogger to do? Is this the greatest study of all time or just a bunch of malarkey? Perhaps it's somewhere in the middle.

For help, I decided to seek out a the most neutral party I could find. I was looking for some folks who were very knowledgeable technically, but who were totally disinterested in any particular outcome. I wanted people with strong mathematical/statistical backgrounds. I wanted folks who knew how unbiased research ought to be conducted. I wanted folks who couldn't care less whether it was a good study or bad. I found them among the postdocs of a certain research university I know. Not in the education department. Not in economics, but pure mathematicians. These are not political people and they are unaware, so far as I know, that there is even a debate over charter schools.

I sent the study with no commentary other than a request for review of Hoxby's 2007 technical report with comments. Here are the early returns, edited to remove any identifying comments:

Sorry its taken me so long to get back to you...I did read through the paper and I have to say I'm not very impressed.

The entire paper is convoluted and its hard for me to decipher exactly what the findings are and what kind of implications are conjectured based on the findings. That aside, the mathematical approach doesn't appear to be very rigorous and there is no clear explanation of the actual mathematical tools employed.

Again, I am not a statistician so I can't argue the validity of their methods because I don't understand them myself. I can, however, tell you that the explanations of
their methodology are poorly written and give the impression that they might not
understand their methods either.

For example, when explaining the variables used in the 'estimating equations' the variable epsilon_i is not defined but is claimed to "remind us of the robust standard errors clustered at the student level". This is not a definition and not the rhetoric of a mathematician or statistician. Further there is no reference to what the "exhaustive set of lottery fixed effects" and other fixed effects are.

Several times there were unsubstantiated statements made such as "we believe that we should be able to match only about 90% of students, plus or minus a few percents" (pg 13), "Class size has an association with achievement effects that is estimated with a fair degree of precision" (pg 35), and "we estimate the magnitude of the understatement to be about 8 percentage points" (pg 15). Why do they believe that a 90% match is 'good enough'? Where did that statistic come from? What is a 'fair degree of precision'? How did they estimate the understatement to be 8 percentage points?

These statements alone are enough for me to discount the entire validity of the paper because if we can identify one statistic that was pulled out of thin air, then why should we believe that all the others aren't as well?

Also, for several results, reasons are given as to why the result should not be regarded as significant. For example, the authors clearly state why policy effects
should be discounted (p35). However, they go on to evaluate policy effects as well as speculate on reasons for the results. If policy effects should so clearly be disregarded, then they shouldn't be analyzed.

Finally, I saw many instances of bias remarks made which should be carefully avoided when evaluating statistics.

So there you have it.

The paper has now been passed on to another department at the same university for a confirming/disconfirming review. I'll let readers know if we learn anything from that.

Fool's Gold from Hoxby

If her study is above reproach,

Why hide the data?

In September, conservative Hoover Institution economist Caroline Hoxby released an update to her 2007 study of charter schools in New York City. Hoxby claimed that compounded gains for an average student continuously enrolled in third to eighth grade in a charter nearly closed the “Harlem-to-Scarsdale” achievement gap and implied that the trend would continue, although that was apparently outside of the scope of her data. Technical report here.

At last, charter school advocates had some good news in a sea of equivocal data. The news was so good that pro-charter political groups began broadcasting Hoxby-as-gospel without pausing for that customary scholarly requirement; a peer review.

Why should they? The Hoover Institution's mission is "collecting knowledge, generating ideas, and disseminating both" in support of "private enterprise"as opposed to "governmental, social or economic action." According to Jim Horn at Schools Matter, it was charter advocates who ordered the study - which one assumes, is what rich politicians do when they don't like the existing studies.

It was a huge embarrassment to Gates, Broad, Bloomberg, and other corporate school charterizers when the first peer-reviewed national study of charter schools showed that charters outperform public schools in less than 20 percent of the cases. Since then the oligarchs have brought in the hired gun economist, Catherine Hoxby, who never saw a charter school she didn't like, to slime the study as best she can.

Bloomberg has even sweetened Hoxby's pot, asking her for a good-news study of New York's charter schools, a study whose methodology Hoxby immodestly refers to as reflecting the "the gold standard."

Princeton's Jesse Rothstein objected to Hoxby's method in an earlier study in a series of back and forth papers where he complains that,

1) Hoxby has not made available the data that were used for her original paper.
2) Hoxby's basic result holds only when we rely on her specific construction of the larger streams variable. She has never presented estimates that do not rely on this variable; all such estimates that I have computed yield small, insignificant effects of choice on test scores.
3) There are several odd aspects of Hoxby.s particular larger streams variable that cast doubt on its validity.
4) There are serious errors in both the program code and data that Hoxby has distributed.
5) Hoxby has not released the code that was used in her Reply, but she offers no indication that these errors have been repaired.

Once Rothstein published that he was unable to replicate Hoxby's study due to its flaws, she was quick to suggest in an interview with the Harvard Crimson, that the dispute was part of a larger issue of “race and gender bias” in higher education. Hoxby is black, while both Rothstein and his dissertation adviser at University of California, Berkeley, Professor David Card, are white.

Hoxby was similarly skewered over the 2009 study by the late Arizona State University research debunker Gerald Bracey, in the Huffington Post. Bracey wrote,

The study was conducted by economist Caroline Hoxby, the only person in the whole country who consistently finds results that favor charters. Here are a few cogent items about the study:

1. Many of the data that would be needed to draw conclusions are not presented.
2. The study is limited to New York City.
3. The study has not been peer reviewed.
4. The study was published by a pro-charter advocacy group staffed with people
   who used to work in charter schools.
5. The editorial writer, Jo-Ann Armao, lacks the background in econometric research to actually know how to interpret the study. She was, therefore, engaging in faith-based editorializing but passing it off as evidence-based.
6. Even if the study proves to be sound (unlikely according to some other researchers who have also looked at it), it is only ONE study. Strong conclusions in any field should never be drawn on the basis of only one study.

Where did all this furor begin? Over at SusanOhianian.org, Bracey tells the story of how certain well-funded private think tanks ignored the evidence (because it disagreed with their ideology) and moved the political playing field from the laboratory to the newspapers.

The "gold standard" Hoxby likes to refer to is the mere existence of a lottery used at most charter schools when applicants outnumber capacity; known as being oversubscribed. But this would never pass for gold standard research in biomedical research.

Hoxby's study is no clinical trial.

The "gold standard" applies to double-blind experimental studies with random assignment. That means that some research participants get the experimental treatment and some get a placebo, and both are assigned randomly. It also means that neither the researchers nor the participants know who is getting which treatment. After all, expectations are important, and the mind can set us up for all kinds of things. The fact that Hoxby's study fails to meet the rigor of the gold standard seems to bother her fans - not at all.

Where is the placebo?

Without a placebo subjects who know they are receiving a treatment may behave differently. On the other hand, what impact might non-selection have on a student denied entry to their desired school?

What about peer effects?

From a personal level, if you have a child, you don't care about controlling for peer effects. Actually, you want the effects to be left in so that you can take advantage of them. If charter schools have "better" students, that's a reason to send your own child to a charter school. However, if this analysis is done for policy purposes, to influence policy-makers, then peer effects do matter. If you are thinking about all students, not just the select few who can get into the "better" school, you need to control for peer effects.

Then there's selection bias on the School Level .

The goal of this lottery-based study design is to avoid self-selection bias in the data. However, those who use it do not acknowledge the additional selection problems they create.The most important problem is that not all charter schools are oversubscribed, so not all charter schools can be included in these studies. This wouldn't be a problem if we had good reason to believe that a random selection of charter schools were included, but that is obviously not the case. Clearly, the "better" charter schools are far, far, far more likely to be oversubscribed than the "worse" charter schools. This biases the sample rather severely towards better charter schools.

And a really strong public school is not going to lose a lot of students to a simply above-average charter school; which introduces another bias toward charters.

Finally generalizability remains a problem.

Lottery-based studies only include the kinds of students and families that apply to charter schools in the first place. Even if the previous issues could be corrected, how can one know that other sorts of students and families would see the same benefits? The fact is that different populations might benefit less or more from going to a charter school. It is simply impossible to know from this kind of study. Of course, if you are only concerned about benefitting the kids of families who already opt for charter schools, then this is not a problem. But if you aim to help a broader population than that, you need a better methodology.

As Bracey put it,

In June, 2009, another study of charters--this one more of a national look-- concluded that 46% of charters did no better than comparable public schools, 17% outperformed the publics and 37% did worse. "We've got two bad charter schools for every good one," said Margaret Raymond, the Stanford University researcher who conducted the study. Raymond has been known as a charter supporter so her willingness not to flinch in the face of these data is admirable. Would the FDA approve a drug that had adverse effects twice as often as positive effects?

In reality, the rigors of double-blind studies would be nearly impossible to adhere to in social sciences, which, for this and other reasons, should never be compared to biomedical research. Social scientists have relatively poor control over their variables; but sometimes like to act as if they do.

On a policy level, we need to be concerned with charters more generally than that. If we ... approve new charter schools, we have to expect an average charter school to result, not an exceptional one.

Hoxby compares test results of students enrolled in NY’s charter schools to the results for those students who applied to a charter but were not selected for admission. This approach shines light on the probability that charters enroll more motivated families.

As the former principal of Cassidy School, I can attest, first-hand, to the power of a unified team of adults (parents and teachers with resources) working cooperatively for the benefit of all of their children. I suspect it is this "motivation" that is the cause of stronger student achievement. To the extend that those motivational forces can be marshaled in any given public, or charter, or private school; I think you get a much better school. I suspect the "charter school effect" is some manifestation of that teamwork by a community of responsible, hard-working adults.

This from EdWize:

Hoxby’s findings are encouraging: by the third grade, the average charter school student was 5.8 points ahead of the lotteried-out counterpart in math and was 5.3 points ahead in English Language Arts. As Hoxby follows students’ achievement from 2001 to 2008, she also finds that the average charter school student gained 3.6 more points each year in math and 2.4 more points each year in ELA. For an average charter student continuously enrolled in grades four through eight, the effect is larger with annual gains of 5.0 points in math and 3.6 in ELA above the performance of the lotteried-out student. (Last year, nine charters enrolled students across all of these grades.)

Such a dramatically-presented conclusion is sure to feature prominently in charter advocates’ efforts to expand the number of charter schools across the city and state. And if it’s true, then why shouldn’t we?

Even if we assume Hoxby's data are correct - a rather large assumption under the circumstances - states ought to be somewhat concerned about resegregation.

Hoxby finds that 92 percent of charter students are black or Hispanic, compared to 72 percent in district schools and concludes that “the existence of charter schools in the city therefore leaves the traditional public schools less black, more white, and more Asian.” Such racial segregation is consistent with research on charter schools in other states including North Carolina, Texas and elsewhere.

Although this statistic is likely to be a function of charter schools’ location in largely black and Hispanic neighborhoods, Hoxby also reports that fewer white students are applying to the charters; although 14 percent of residents in the charter school
neighborhoods are white non-Hispanic, only 4 percent are applying.

Additionally, Hoxby confirms that the city’s charters enroll fewer English language learners (4 percent) than district public schools (14 percent) — despite their location in Hispanic communities.

On another key measure of economic diversity, it’s puzzling that Hoxby compares charter school free- and reduced-price lunch participation rates, which she pegs at 91 percent, to the citywide rate of 72 percent. For a report that prides itself on apples-to-apples comparisons, this charter-to-city comparison is a glaring inconsistency. As Edwize has demonstrated here and here, a better comparison is between charters and the surrounding district/feeder schools. By this comparison, charters have fewer students eligible for free lunch, suggesting that the city’s charters are more affluent than the neighboring district schools.

It’s also worth remembering that charter schools do not have attendance zones and can, in theory, draw students from across neighborhoods. But as the demographic statistics reveal, if charters are attracting applicants from beyond their immediate neighborhoods, it is not generating more racial, ethnic, and economic diversity.

Incidentally, Hoxby’s updated report makes no mention of student attrition from charter schools to district schools, despite the fact that she has access to student-level data over many years.

Presumably she can, and should, confirm or reject the anecdotal reports of this occurrence, as student attrition biases achievement results. Attrition is also a measure of equity, as district schools must educate all of the students who come through their doors while charters, with site-specific discipline codes and the power to expel students, do not.

Moreover, the report provides tacit evidence that all of the students and families who apply for a charter are in fact different from the other public school parents and students who don’t apply. Whether they are more motivated, place a higher emphasis on education, or some other unmeasurable characteristic, Hoxby finds that “lotteried-out students’ performance does improve and is better than the norm in the U.S. where, as a rule, disadvantaged students fall further behind as they age.”

Given this and the other demographic findings, it is unclear how the report “lays to rest the persistent myth that charter schools “cream” the best students,” as one charter advocate maintains. Quite the opposite — the achievement results for all charter applicants suggests there is likely to be some truth to the creaming “myth.”

Hoxby's data ought to give pause to those charter advocates who are concerned with issues of social justice. Any system that resegregates school children by race is suspect. It is not a satisfactory explanation that such resegregation is being caused by parental choice; segregation has always been caused by parental choice. It is not conclusive that charters have that effect, but Hoxby went some distance showing that it just might.

Hoxby maintains that her method is the “gold-standard” approach when seeking to determine the unique impact of charter school effects, but EdWize questions "two complications that would otherwise moderate her findings."

First, it is conceivable that low-performing charters are undersubscribed and may not have a large “lotteried-out” population to study. In effect, this could under-weight the results of poor performing charters and biases her data in favor of higher-performing charters. A technical version of the paper or scholarly access to her data could ascertain if this is in fact occurring. And despite the reports sweeping conclusions, there are low-performing charters in her study: Hoxby finds that 31 percent of students attend a charter that is performing no better than or worse than the schools attended by lotteried-out counterparts in math and 24 percent are doing no better or worse in ELA.

Second, it’s not clear that Hoxby’s methodology takes into account peer effects — the straightforward concept that students not only learn from their teachers but are also influenced by their peers. Charter schools benefit from the fact that 100 percent of their students hail from motivated families; as a result, a charter student is surrounded by peers who are there by choice — rather than by attendance zone. In contrast, the “lotteried-out” students may not benefit from such an intensive peer effect...

This from Equity Overlooked - Charter Schools and Civil Rights Policy:

The expansion of charter schools is a central policy focus of the Obama administration. Charter schools encompass a variety of schools with different priorities serving many communities and students from a range of backgrounds. There are outstanding and diverse charters, some of which have been highlighted
by this or prior administrations.

While the administration has acknowledged the importance of regulating and closing low performing charter schools, it has yet to respond to concerns raised about continued racial isolation in charter schools. Why is this lack of civil rights oversight so troubling? Without necessary safeguards against the segregating effects of charter schools, disadvantaged families are left to comprehend and cope individually with the complicated landscape of school choice.

Access to the educational marketplace is unequally constrained by a number of factors, including contact with advantaged social networks through which information regarding school quality is exchanged, language barriers, socioeconomic status and the ability of parents to arrange transportation for their schoolchildren. Unless proactive equity measures – like extensive outreach and free transportation – are embedded in the design of charters, and subsequently monitored and enforced, this popular version of education reform simply reinforces unequal educational opportunity.

In the 61-page "Charter Schools' Performance and Accountability: A Disconnect," Bracey summarized the research of others, not doing any original work--found a number of studies showing poor charter school performance.

Diane Ravitch, who served as the Education Department’s assistant secretary for educational research and improvement under President George H.W. Bush, doubts the evidence upon which two of the Obama administration's education efforts are based.

“What is extraordinary about these regulations is that they have no credible basis in research. They just happen to be the programs and approaches favored by the people in power,” writes Diane Ravitch, a research professor of education at New York University, in her blog, Bridging Differences, which is hosted by edweek.org.

In the meantime, a new 2009 study of Baltimore charter schools concluded:

• The number of charter schools increased from 12 in 2005-06 to 22 in 2007-08
• Enrollment in charter schools increased from 2,925 students in 2005-06 to 5,520 in 2007-08, an increase of almost 89%.
• There is significant variation among charter schools on most measures of school and student performance.
• On average, charter school students are less disadvantaged than non-charter school students. Charter school students are less likely to be eligible for special education services, to be over-age for their grade, or to be FARMS-eligible.
• Overall, charter schools do not enroll a higher proportion of students from outside the district or from non-public schools than non-charter schools. There is, however, significant variation among individual schools in the enrollment of students new to City Schools, with selected schools enrolling a larger proportion of students from outside the district.
• Some charter schools attract a significant proportion of their enrollment from surrounding non-charter City schools.
• Charter school students are less likely to leave the district at the end of the school year and are more likely to re-enroll in their schools the following school year compared with non-charter school students.

The fact is the evidence on charter schools is not in. Some are great. Some are terrible. Most are somewhere in the middle. Just like public schools.

Charters outperform public non-charters is some places (apparently both Hoxby and CREDO agree that New York is one such place) and the reverse is true elsewhere.

The reason for the performance of students in the best schools (of any variety) seems less likely to be due to anything specific to charters. Rather, it is more likely to be due to site-specific conditions of parental and teacher motivation to build a good school.

The amount of time students spend engaged in academic work also seems to be central to school success. In some cases, under binding arbitration, union restrictions on the amount of time a teacher may be asked to work, might play a role.

Political think tanks that promote Hoxby's study as definitive are suspect for political and ideological motivation and provide strong evidence that they are not truly research facilities.

All this being said, Kentucky still has several schools that have languished in tier 4 and 5 purgatory for some time now. We must assume that the school districts where these schools exist, have faithfully tried, but failed to make a difference for the children under their care. In such cases, resisting a fresh new approach - even a privately run approach like charters - suggests a greater devotion to the feelings of adults than the needs of students. Kentucky needs a strong, but limited, charter law that continues the effort to produce a high-quality school in every community.

Hoxby: Too early to draw sweeping conclusions on charters

In the wake of KET's recent discussion of charter schools the communication lines aroud here have perked up.

Some have asked me to look into Sheldon Berman's statements, and I will - next chance I get. It is too soon to draw conclusions about charters. That includes calling charter schools a failed reform. That is nonsense. Just like sclling charter schools a civil right is nonsense.

One KSN&C reader wanted to know why I remain unconvinced by Carolyn Hoxby's research which suggests that New York's charter school students outperform their public school peers. The reader asked why I don't love her method of comparing charter school students with nearby public school students.

Well, in part, it's because I'm not sure they are peers. The other part is an admitted suspicion of psuedoscientific work from some institutions that collect (buy) like-minded researchers and then set out to create data in support of their predetermined worldview.

I have no idea how to guard against this - except to remain suspicious, think critically, and to wait for scholarly confirmation through peer review before jumping to any conclusions.

Speaking of which, there's this from "Advantage None: Re-Examining Hoxby’s Finding of Charter School Benefits" by Lawrence Mishel and Joydeep Roy on Hoxby's 2004 study:

Hoxby's analysis, however, suffers from the fact that her method of comparing charter schools to their neighboring regular public schools (and to those neighboring public schools with a similar racial composition) inadequately controls for student backgrounds. In her sample of matched schools there are often significant differences in the demographic and socioeconomic characteristics of the students. For instance, comparing the charter schools in Hoxby's sample to the matched neighboring public schools with a similar racial composition shows that the charter schools have a disproportionately higher black population (34% vs. 28%) and higher white population (43% vs. 36%), while the share of Hispanics is lower (18% vs. 30%). Her sample of charter schools also has disproportionately fewer low-income students than does the matched "racially similar" sample of neighboring public schools (49% vs. 60%). The same picture emerges in terms of the demographics of charter schools in central cities: charter schools serve a disproportionately lower share of minorities and low-income students compared to their matched regular public schools. Thus,
without further controls, Hoxby's method of comparing "racially matched" schools does not appear to be effective in controlling for student characteristics.

Hoxby's result of a positive charter effect on math proficiency disappears when racial composition is controlled for directly. Further, when both racial composition and low-income status are controlled for, the positive effect of attending a charter school disappears for both math and reading (it becomes very small and not statistically significantly different than zero). Thus, Hoxby's conclusion that "although it is too early to draw sweeping conclusions, the initial indications are that the average student attending a charter school has higher achievement than he or she otherwise would" (Hoxby 2004a) does not hold up against direct controls for student background.

Plain and simple - the jury is still out on the performance of charter school students relative to public school students.

Now for those who want me to quit doubting Hoxby and start lovin' her findings, particularly in the wake of CREDO's recent softening, my answer is - I can't. At least not yet. Let's have some pros give her data the twice over before we start promising the public something we cannot deliver.

In the meantime, surely skeptics and hopeful romantics alike can agree with Hoxby when she says it's "too early to draw sweeping conclusions."

Charter Debate to Continue on KET

Charter schools will be debated on

KET’s “Kentucky Tonight” show

at 8 p.m. Monday, January 18.

Scheduled guests include:

• Rev. Jerry Stephenson, chair of the Kentucky Education Restoration Alliance
• Jim Waters, director of policy and communications for the Bluegrass Institute for Public Policy Solutions
• Sharron Oxendine, president of the Kentucky Education Association
• Superintendent Sheldon Berman of Jefferson County Public Schools

Charter schools are public schools contracted out to the private sector. In 1992, the first two charter schools operated in St. Paul, Minnesota. By September 1999, almost 300,000 students attended 1,682 charter schools operating in 33 states. By 2000, 38 states had laws allowing charter schools and a year later there were 2,372 such schools in America.

More recently the heat has been turned up on states lacking permissive charter school laws because Education Secretary Arne Duncan tied billions of federal dollars to state acquiescence. Pass a charter school law and better your chances in the Race To The Top sweepstakes.

WFPL reported that several Louisville pastors have been working with Republican Rep. Brad Montell of Shelbyville who filed a bill to allow charter schools in the state. Rep. Stan Lee of Lexington filed a similar bill in July. Both would have permitted charters to operate in competition with successful schools. The vote on Montell's bill was tied in committee, so lacking a majority, died. But supporters sense they are close and vow to push on.

• Expect Stephenson to say that the greatest need for charter schools is in inner city Louisville where a number of schools are failing.
• Expect Waters, the BIPPS communications director, to swear by Carolyn Hoxby's widely discredited study and offer it as proof of charter school effectiveness despite substantial evidence that charter performance is as varied as that of the public schools they would replace.
• Expect Superintendent Berman to defend JCPS's continued efforts to turn around their most challenging schools despite years of stagnant results.
• Expect Oxendine to question the need for charter schools based on their mixed performance nationally.

The data on charter schools is far from conclusive. Taken as a whole it's impossible to conclude that they are any better, and are sometimes worse, than the public schools. But in places, they have shown success. As I have said before, if education were a natural science, it would be like meterology; highly-localized and ever-changing. Successful charter schools seem to focus on the success of each child, building relationships, and a high quality faculty working their butts off - like in successful public schools.

As early as 1999, Arizona researchers found "evidence of substantial ethnic segregation," and that charter schools "were higher in white enrollment than other public schools." (Cobb & Glass)

In "Does Choice Lead to Racially Distinctive Schools?" Weiher and Tedin (2002) found "that race is a good predictor" of the school choice families make. Whites, African Americans, and Latinos transfer into charter schools where their groups comprise between 11 and 14 percentage points more of the student body than the traditional public schools they are leaving.

In "Decade of Charter Schools: From Theory to Practice, Bulkley and Fisler (2003) at Rutgers
found that "although some successes are evident, there is still much to learn about the quality of charter schools and the experiences of charter school stakeholders. There is strong evidence that parents and students who remain in charter schools are satisfied and that charter schools are more autonomous than other public schools. But the jury is still out on some of the most important questions, including those about innovation, accountability, equity, and outcomes."

In "The effect of charter schools on charter students and public schools ," Bettinger (2004) at Case Western Reserve found that "test scores of charter school students do not improve, and may actually decline, relative to those of public school students," but charters had no significant effect on test scores in neighboring public schools.

In "Charter Schools and Student Achievement in Florida," Florida State's Tim Sass (2006) found achievement to initially be lower in charters. However, by their fifth year of operation charter schools "reach a par with the average traditional public school." Charters targeting at-risk and special education students demonstrate lower student achievement, as do their public counterparts. He found no difference between charters managed by for-profit entities than charters run by nonprofits.

In "Skimming the Cream" West, Ingram & Hind (2006) of the London School of Economics and Political Science found evidence suggestive of both "cream skimming" and "cropping off" educational provision to particular groups of students. "It is concluded that the introduction of market oriented reforms into public school systems requires monitoring and effective regulation to ensure that autonomous schools do not act in their own self-interest."

In "The Charter School Allure: Can Traditional Schools Measure Up? Bowling Green State's May (2006) found that "urban school districts are losing significant resources to charter schools" and that "despite the lack of statistically significant evidence of academic gains, parents perceive an enhanced educational experience." The author surmises that the chasm between perceived charter school success and traditional school failure is a "perception gap"

In "The Impacts of Charter Schools on Student Achievement: Evidence from North Carolina," Bifulco at U Conn and Duke's Ladd (2006) found that "students make considerably smaller achievement gains in charter schools than they would have in public schools" and say there is "suggestive evidence" that "about 30 percent of the negative effect of charter schools is attributable to high rates of student turnover."

Then came the study that defined the debate.

It was the first peer-reviewed detailed national assessment of charter school impacts since its longitudinal, student-level analysis covers more than 70 percent of the nation’s students attending charter schools. In “Multiple Choice: Charter School Performance in 16 States,” reasearchers at the Center for Research on Education Outcomes (CREDO) at Stanford University found that "there is a wide variance in the quality of the nation’s several thousand charter schools with, in the aggregate, students in charter schools not faring as well as students in traditional public schools."

While the report recognized a robust national demand for more charter schools from parents and local communities, it found that 17 percent of charter schools reported academic gains that were significantly better than traditional public schools, while 37 percent of charter schools showed gains that were worse than their traditional public school counterparts, with 46 percent of charter schools demonstrating no significant difference.

Then, something unusual happened.

Education Week reported Stanford colleague Caroline M. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City. That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found.

In a memorandum titled "Fact vs. Fiction: An Analysis of Dr. Hoxby’s Misrepresentation of CREDO’s Research," the Center for Research on Education Outcomes, (CREDO), fired back.

The memo, "A Serious Statistical Mistake in the CREDO Study of Charter Schools," by Caroline Hoxby, does not provide any basis whatsoever for discounting the reliability of the CREDO study’s conclusions. The central element of Dr. Hoxby’s critique is a statistical argument that is quite unrelated to the CREDO analysis. The numerical elements of it are misleading in the extreme, even had the supporting logic been correct. Unfortunately, the memo is riddled with serious errors both in the structure of the underlying statistical models and in the derivation of any bias.

This is all going on at Stanford University, the same campus where the conservative Hoover Institution does its work with such notable conservatives as Condolesa Rice and Donald Rumsfeld. It the same place where Eric Hanushek tries to prove that money doesn't matter in education. One gets the distincitive au de political bias from the place. Turns out that Hoxby works there too. What a surprise.

“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said associate professor of education and sociology Sean F. Reardon, a colleague of both scholars at Stanford told Ed Week. “But I don’t think either side got it right,” he added.

Neither study has been published yet in a peer-reviewed journal. But that won't stop political operatives from citing them as definitive evidence.