Saturday, March 24, 2012

AJC Uncovers Suspicious Test Scores Across the Nation

This from the Atlanta Journal-Constitution:
Suspicious test scores in roughly 200 school districts resemble those that entangled Atlanta in the biggest cheating scandal in American history, an investigation by The Atlanta Journal-Constitution shows.

The newspaper analyzed test results for 69,000 public schools and found high concentrations of suspect math or reading scores in school systems from coast to coast. The findings represent an unprecedented examination of the integrity of school testing.

The analysis doesn't prove cheating. But it reveals that test scores in hundreds of cities followed a pattern that, in Atlanta, indicated cheating in multiple schools.

A tainted and largely unpoliced universe of untrustworthy test results underlies bold changes in education policy, the findings show. The tougher teacher evaluations many states are rolling out, for instance, place more weight than ever on tests.

Perhaps more important, the analysis suggests a broad betrayal of schoolchildren across the nation. As Atlanta learned after cheating was uncovered in half its elementary and middle schools last year, falsified test results deny struggling students access to extra help to which they are entitled, and erode confidence in a vital public institution.

"These findings are concerning," U.S. Secretary of Education Arne Duncan said in an emailed statement after being briefed on the AJC's analysis.

He added: "States, districts, schools and testing companies should have sensible safeguards in place to ensure tests accurately reflect student learning."

In nine districts, scores careened so unpredictably that the odds of such dramatic shifts occurring without an intervention such as tampering were worse than one in 1 billion.

In Houston, for instance, test results for entire grades of students jumped two, three or more times the amount expected in one year, the analysis shows. When children moved to a new grade the next year, their scores plummeted — a finding that suggests the gains were not due to learning.

Overall, 196 of the nation's 3,125 largest school districts had enough suspect tests that the odds of the results occurring by chance alone were worse than one in 1,000.

For 33 of those districts, the odds were worse than one in a million.

A few of the districts already face accusations of cheating. But in most, no one has challenged the scores in a broad, public way...

Both critics and supporters of testing said the newspaper's findings are further evidence that in the frenzy to raise scores, the nation failed to pay enough attention to what was driving the gains.

"We are putting way too much pressure on people to raise scores at a very large clip without holding them accountable for how they are doing it," said Daniel Koretz, a Harvard Graduate School of Education testing expert.

Test-score pressure is palpable in schools grappling with urban blight and poverty.
These are the schools that the 2001 No Child Left Behind Act was supposed to fix...

The findings call into question the approach that dominated federal education policy over the past decade: Set a continuously rising bar and leave schools and districts essentially alone to figure out how to surmount it — or face penalties.

"If you want to keep your job, keep your school out of the news, keep winning awards and advance in your career, you need to make your school look better," said Joseph Hawkins, a former testing official with the Montgomery County, Md., school system.

Koretz, the Harvard expert, said cheating is one extreme on a continuum that, at its other end, includes gaming the test in legal ways — such as through test-prep drills — that don't significantly increase students' overall knowledge or skills.

Even as state test scores have soared, students' performance on national and international exams has been more mediocre. Cheating and gaming may help explain why....

In some of the nation's biggest cities, dynamic district leaders preached "data-driven" decision-making and even linked test scores to bonuses or principal hiring and firing decisions. Many boasted of taking a corporate approach to education, focusing on student test achievement as the single most important measure of success.

Some of the most persistently suspicious test scores nationwide, however, occurred in districts renowned for cutting-edge reforms...

In 2002, Houston was the first winner of the Broad Prize, which has become the most coveted award in urban education. The Eli and Edythe Broad Foundation praised Houston's intense focus on test results. More recently, Houston has been among the leaders in tying teacher pay to student test scores.

But twice in the past seven years, the AJC found, Houston exhibited fluctuations with virtually no chance of occurring except through tampering...

Critics of testing have complained for years that increased pressure brought on by accountability measures leads to more testing abuses.

Education historian and New York University Professor Diane Ravitch said the incessant focus on testing has eroded the quality of instruction.

"All of this is predictable," said Ravitch, a former top U.S. Department of Education official who in recent years reversed her support for testing and tough accountability measures. "We're warping the education system in order to meet artificial targets."

Through programs such as Race to the Top, federal education officials have pushed states to adopt more aggressive teacher evaluation systems that, typically, consider test scores.

"Whatever the stakes were under No Child Left Behind," Ravitch said, "they are going to be much higher, now that teachers are being told your scores are going to be public and you're going to be fired if they don't go up X number of years in a row."...
Here's who's on the list from Kentucky (Map): 

StateDistrict NameDescendingCity% flagged 2009% flagged 2009% flagged 2010% flagged 2011
KYCovington Independent\N03.5704.17Details   
KYClay CountyMANCHESTER022.7313.6416.67Details   
KYClark CountyWINCHESTER2.52.500Details   
KYChristian CountyHOPKINSVILLE6.253.857.6911.54Details   
KYCarter CountyGRAYSON0000Details   
KYBullitt CountySHEPHERDSVILLE5.714.292.787.14Details   
KYBreckinridge County\N0000Details   
KYBreathitt CountyJACKSON508.339.09Details   
KYBoyd CountyASHLAND0005Details   
KYBowling Green IndependentBOWLING GREEN0008.33Details   
KYBoone CountyFLORENCE5.882.944.419.68Details   
KYBell CountyPINEVILLE1.721.7200Details   
KYBarren CountyGLASGOW05.2602.38Details   
KYAshland IndependentASHLAND002.780Details   
KY\NWEST LIBERTY0000Details   
KY\NVERSAILLES0050Details   
KY\NERLANGER0005Details   
KY\NWILLIAMSTO0050Details   
KY\NLIBERTY05510Details   
KY\NPIKEVILLE010.915.777.14Details   
KY\NALEXANDRIA029.1700Details   
StateDistrict NameDescendingCity% flagged 2009% flagged 2009% flagged 2010% flagged 2011
KYLetcher CountyWHITESBURG6.259.381.617.81Details   
KYLeslie CountyHYDEN0000Details   
KYLawrence CountyLOUISA7.6911.5400Details   
KYLaurel CountyLONDON9.523.853.853.85Details   
KYKnox CountyBARBOURVILL9.0902.172.17Details   
KYKnott CountyHINDMAN003.5710.71Details   
KYKenton CountyERLANGER3.336.6751.67Details   
KYJohnson CountyPAINTSVILLE7.8905.5610.53Details   
KYJessamine CountyNICHOLASVILLE10.7110.7103.57Details   
KYJefferson CountyLOUISVILLE8.688.114.175.24Details   
KYHopkins CountyMADISONVI8626Details   
KYHenderson CountyHENDERSON2.552.55Details   
KYHart CountyMUNFORDVILLE8.33002Details   
KYHarrison CountyCYNTHIANA0000Details   
KYHarlan CountyHARLAN5.137.695.132.7Details   
KYHardin CountyELIZABE4.69000Details   
KYGreenup CountyGREENUP004.174.17Details   
KYGrayson CountyLEITCHFIELD10550Details   
KYGraves CountyMAYFIELD8800Details   
KYFranklin CountyFRANKFORT0000Details   
KYFloyd CountyPRESTONS11.296.673.338.06Details   
KYFleming CountyFLEMINGSBURG0005Details   
KYFayette CountyLEXINGTON3.987.560.562.75Details   
KYElliott CountySANDY HOOK10000Details   
KYDaviess CountyOWENSBORO3.337.141.671.67Details   

StateDistrict NameDescendingCity% flagged 2009% flagged 2009% flagged 2010% flagged 2011
KYWolfe CountyCAMPTON4.552000Details   
KYWhitley CountyWILLIAMSBURG2.785.267.8910.81Details   
KYWebster CountyDIXON2.08002.08Details   
KYWashington County\N0000Details   
KYWarren CountyBOWLING GRE2.72.3803.66Details   
KYShelby CountySHELBYVILLE03.133.136.25Details   
KYScott CountyGEORGETOWN2.5000Details   
KYRussell CountyJAMESTOWN03.853.850Details   
KYRowan County\N0000Details   
KYPulaski CountySOMERSET57.57.55.88Details   
KYPerry CountyHAZARD51.115.957.61Details   
KYOldham CountyBUCKNER5.7701.795.36Details   
KYOhio CountyHARTFORD0000Details   
KYNelson CountyBARDSTOWN2.632.6352.5Details   
KYMuhlenberg CountyGREENVILLE3.573.573.5710.71Details   
KYMeade CountyBRANDENBURG2.782.788.823.13Details   
KYMcCracken CountyPADUCAH08.335.565.56Details   
KYMartin CountyINEZ0000Details   
KYMarshall CountyBENTON5.562.788.330Details   
KYMarion CountyLEBANON004.170Details   
KYMagoffin CountySALYERSVILLE2515010Details   
KYMadison CountyRICHMOND6.673.335.363.13Details   
KYLogan CountyRUSSELLVILLE0004Details   
KYLincoln CountySTANFORD7.143.5707.14Details   
KYLewis County\N5005Details
A class is a group of students in the same school from one year to the next. For example, fourth grade students in 2009 and fifth grade students in 2010. A "flag" only indicates a test-score shift outside the norm.

Clicking on "Details" reveal this type of data:

District nameFayette County
Street address701 EAST MAIN ST
CityLEXINGTON
2008
Number of classes176
Percentage of classes flagged3.98
2009
Number of classes172
Percentage of classes flagged7.56
2010
Number of classes180
Percentage of classes flagged0.56
2011
Number of classes182
Percentage of classes flagged2.75
And who has the largest percentage of flagged classes in Kentucky? 
Well, it looks like...

District nameClay County
Street address128 RICHMOND RD
CityMANCHESTER
2008
Number of classes26
Percentage of classes flagged0
2009
Number of classes44
Percentage of classes flagged22.73
2010
Number of classes44
Percentage of classes flagged13.64
2011
Number of classes42
Percentage of classes flagged16.67

Only one central Kentucky district has more than 10% of classes flagged:

District nameJessamine County
Street address501 EAST MAPLE
CityNICHOLASVILLE
2008
Number of classes28
Percentage of classes flagged10.71
2009
Number of classes28
Percentage of classes flagged10.71
2010
Number of classes28
Percentage of classes flagged0
2011
Number of classes28
Percentage of classes flagged3.57
Analysis Limitations from AJC: A statistical analysis cannot prove cheating. It can only identify improbable events that can be caused by cheating and should be investigated. Ideally we would look at how individual student test scores change from year to year, but federal privacy regulations precluded access to that data. The approximate cohorts we used are an imperfect substitute. It is unlikely that two groups of students in a cohort are perfectly identical. Urban districts in particular have high student mobility. But we found that large demographic changes at a school - large increases or decreases in poverty levels, for example - are rare. Approximate cohorts mostly compare similar students. Because of this, large jumps or dives in test scores should be rare, experts told us. Matching schools and grades between years in this way makes it impossible to compare groups of students who change schools.
Posted by at

4 comments:

Anonymous said...

Wow!

This is a huge shot at educator credibility.

Richard, do you think the AJC methodology is credible?

March 25, 2012 at 8:01 PM
Anonymous said...

If your scores go up then you are cheating. If your scores stay low or go down, then you are failing. Seems like the ideal situation is to struggle to make slow progress - wow, that's were I want my kids to go or to spend my career.

If you flip the data, it seems to also say that 90% of the schools' scores are were they should be.

March 26, 2012 at 8:13 AM
Richard Day said...

As AJC states, this analysis cannot prove cheating. But it is suggestive of some concern. When AJC dug deeper into Atlanta's scores, that's when folks started confessing and the scandal grew.

And yes, I am sure most folks do it right and some cheat.

March 26, 2012 at 8:09 PM
Anonymous said...

Wonder how much more math a few kids could have learned if the resources to create this report were used to spend direct time with students.

April 10, 2012 at 9:24 AM

Post a Comment

Subscribe to: Post Comments (Atom)