Test Scores Often Misused in Policy Decisions

This from the Huffington Post:

Education policies that affect millions of students have long been tied to test scores, but a new paper suggests those scores are regularly misinterpreted.
According to the new research out of Mathematica, a statistical research group, the comparisons sometimes used to judge school performance are more indicative of demographic change than actual learning.

For example: Last week's release of National Assessment of Educational Progress scores led to much finger-pointing about what's working and what isn't in education reform. But according to Mathematica, policy assessments based on raw test data is extremely misleading -- especially because year-to-year comparisons measure different groups of students.

"Every time the NAEP results come out, you see a whole slew of headlines that make you slap your forehead," said Steven Glazerman, an author of the paper and a senior fellow at Mathematica. "You draw all the wrong conclusions over whether some school or district was effective or ineffective based on comparisons that can't be indicators of those changes."

"We had a lot of big changes in DC in 2007," Glazerman continued. "People are trying to render judgments of Michelle Rhee based on the NAEP. That's comparing people who are in the eighth grade in 2010 vs. kids who were in the eighth grade a few years ago. The argument is that this tells you nothing about whether the DC Public Schools were more or less effective. It tells you about the demographic."
Those faulty comparisons, Glazerman said, were obvious to him back in 2001, when he originally wrote the paper. But Glazerman shelved it then because he thought the upcoming implementation of the federal No Child Left Behind act would make it obsolete.

That expectation turned out to be wrong. NCLB, the country's sweeping education law which has been up for authorization since 2007, mandated regular standardized testing in reading and math and punished schools based on those scores. As Glazerman and his coauthor Liz Potamites wrote, severe and correctable errors in the measurement of student performance are often used to make critical education policy decisions associated with the law.

"It made me realize somebody still needs to make these arguments against successive cohort indicators," Glazerman said, referring to the measurement of growth derived from changes in score averages or proficiency rates in the same grade over time. "That's what brought this about." So he picked up the paper again.

NCLB requires states to report on school status through a method known as "Adequate Yearly Progress." It is widely acknowledged that AYP is so ill-defined that it has depicted an overly broad swath of schools as "failing," making it difficult for states to distinguish truly underperforming schools. Glazerman's paper argues NCLB's methods for targeting failing schools are prone to error.

"Don't compare this year's fifth graders with last year's," Glazerman said. "Don't use the NAEP to measure short-term impacts of policies or schools."

The errors primarily stem from looking at the percentage of students proficient in a given subject from one year to the next -- but it measures different groups of students from year to year, leading to false impressions of growth or loss.

Hat tip to the Commish.

If We Tested School Board Members...

Over the years, finding a Kentucky school board member (or Trustee) who wasn't very smart was a fairly simple task. Well-educated Trustees were apparently the exception. But things have changed over the decades, and today's school board members are generally among the better educated citizens in most communities.

But what would happen if they had to be tested the same way students are - say, by taking the 10th grade exams? Well, that's never going to happen, right?

In Florida, it did.

This from Marion Brady in the Answer Sheet and here:

When an adult took standardized tests forced on kids

A longtime friend on the school board of one of the largest school systems in America did something that few public servants are willing to do. He took versions of his state’s high-stakes standardized math and reading tests for 10th graders, and said he’d make his scores public.

By any reasonable measure, my friend is a success. His now-grown kids are well-educated. He has a big house in a good part of town. Paid-for condo in the Caribbean. Influential friends. Lots of frequent flyer miles. Enough time of his own to give serious attention to his school board responsibilities. The margins of his electoral wins and his good relationships with administrators and teachers testify to his openness to dialogue and willingness to listen.

He called me the morning he took the test to say he was sure he hadn’t done well, but had to wait for the results.

Turns out the board member was quite a fella.

The man in question is Rick Roach, who is in his fourth four-year term representing District 3 on the Board of Education in Orange County, Fl., a public school system with 180,000 students. Roach took a version of the Florida Comprehensive Assessment Test, commonly known as the FCAT, earlier this year...

Roach, the father of five children and grandfather of two, was a teacher, counselor and coach in Orange County for 14 years. He was first elected to the board in 1998 and has been reelected three times. A resident of Orange County for three decades, he has a bachelor of science degree in education and two masters degrees: in education and educational psychology. He has trained over 18,000 educators in classroom management and course delivery skills in six eastern states over the last 25 years....

Now in his 13th year on the board, he had considered taking the test for a while as he began to increasingly question whether the results really reflected a student’s ability. He was finally pushed to do it earlier this year, he said, after a board meeting at which the chairman listed five goals, and one of them caught his attention for being so unremarkable.

Roach said: ‘He [the chairman] said that by 2013 or 2014, he wanted 50 percent of the 10th graders reading at grade level....I’m thinking, ‘That’s horrible.’ Right now it’s 39 percent of our kids reading at grade level in 10th grade. I have to tell you that I’ve never believed that that many kids can’t read at that level. Never ever believed it. I have five kids of my own. None of them were superstars at school but they could read well, and these kids today can read too.

“So I was thinking, ‘What are they taking that tells them they can’t read? What is this test? Our kids do okay on the eighth grade test and on the fifth grade test and then they get stupid in the 10th grade?”...

Here's his take on the experience.

“I won’t beat around the bush. The math section had 60 questions. I knew the answers to none of them, but managed to guess ten out of the 60 correctly. On the reading test, I got 62% . In our system, that’s a ‘D,’ and would get me a mandatory assignment to a double block of reading instruction.

“It seems to me something is seriously wrong. I have a bachelor of science degree, two masters degrees, and 15 credit hours toward a doctorate. I help oversee an organization with 22,000 employees and a $3 billion operations and capital budget, and am able to make sense of complex data related to those responsibilities....

“It might be argued that I’ve been out of school too long, that if I’d actuall y been in the 10th grade prior to taking the test, the material would have been fresh. But doesn’t that miss the point? A test that can determine a student’s future life chances should surely relate in some practical way to the requirements of life. I can’t see how that could possibly be true of the test I took.”

Testing Scandal in Atlanta: The Sum of All Fears

Superintendent Insulated from Teachers

General Counsel's "main duty" - Provide Supt deniability

"A culture of fear and a conspiracy of silence"
infected schools and intimidated teachers

Top Administrators Refuse Responsibility
for anything but success

Criminal charges against educators are possible

District Punished Whistleblower while Guilty went Free

Teacher with low scores forced to crawl under table at faculty meeting
District Violated Open Records, Illegally Altered documents

Test Erasures 3 Standard Deviations above Norm

This from Atlanta 11 Alive:

ATLANTA -- A state investigation into allegations of cheating by Atlanta Public Schools officials on standardized tests finds nearly 80 percent of schools examined cheated on the Criterion-Referenced Competency Tests (CRCT).

An outline of the findings of the investigation from the governor's office:

* Thousands of children were harmed by the 2009 CRCT cheating by being denied remedial education because of their inflated CRCT scores.

* We found cheating in 44 of the 56 schools we examined (78.6%). There were 38 principals of those 56 schools (67.9%) found to be responsible for, or directly involved in, cheating.

* We determined that 178 teachers and principals in the Atlanta Public Schools System cheated. Of the 178, 82 confessed to this misconduct. Six principals refused to answer our questions, and pled the Fifth Amendment, which, under civil law is an implied admission of wrongdoing. These principals, and 32 more, either were involved with, or should have known that, there was test cheating in their schools.

* We empathize with those educators who felt they were pressured to cheat and commend those who were willing to tell us the truth regarding their misconduct. However, this report is not meant to excuse their ethical failings, or exonerate them from their wrongdoings.

* The 2009 CRCT statistics are overwhelming and allow for no conclusion other than widespread cheating in APS. The BRC expert, Dr. John Fremer, wrote an op-ed article for the AJC in which he said there was widespread, organized cheating in APS.

* The drop in 2010 CRCT erasures confirm the conclusion above.

* Cheating occurred as early as 2001.

* There were warnings of cheating on CRCT as early as December 2005/January 2006. The warnings were significant and clear and were ignored.

* Cheating was caused by a number of factors but primarily by the pressure to meet targets in the data-driven environment.

* There was a major failure of leadership throughout APS with regard to the ethical administration of the 2009 CRCT.

* A culture of fear, intimidation and retaliation existed in APS, which created a conspiracy of silence and deniability with respect to standardized test misconduct.

* In addition to the 2009 CRCT cheating, we found other improper conduct: several open record act violations; instances of false statements; and instances of document destruction.  

(Full Report)
"We found cheating in 44 of the 56 schools," Gov. Nathan Deal said as he read from the CRCT report summary during a news conference at the State Capitol Tuesday morning.

"There were 38 principals found to be responsible for or directly involved in cheating," Deal said. In all, he said investigators found 178 teachers and principals were involved with the cheating. Among them, 82 confessed to misconduct and 6 principals refused to participate in the investigation by pleading the 5th Amendment, according to the report.

Recently retired APS Superintendent Dr. Beverly Hall released a statement late Tuesday afternoon denying that she or her staff knew or "should have known" there was widespread cheating.

Hall's statement read as follows:

Having left the Atlanta Public Schools in the very capable hands of Interim Superintendent Erroll Davis, Dr. Beverly Hall has not been given an opportunity to review the Investigators' report to Governor Deal on alleged cheating on the 2009 CRCT. Nor has she been briefed on the contents of the report. Whatever the report may say, Dr. Hall steadfastly denies that she, her staff, or the vast majority of APS teaching and administrative professionals knew or should have known of any allegedly widespread cheating on APS CRCTs in 2009 or any other year. She further denies any other allegations of knowing and deliberate wrongdoing on her part or on the part of her senior staff, whether during the course of the Investigation or before the Investigation began.

Cheating on the CRCT can never be condoned. But whatever cheating may have occurred in the past, test scores in 2010 and 2011 show that APS rests on a firm educational foundation. Tests in those years were conducted under the most stringent security protocols, with State monitors employed in many schools. To be sure, some grades showed declines in 2010. But others showed advances.

Systemwide, scores followed the same upward trend that began in the early 2000s. Scores on this year's CRCT confirmed that trend, with gains over 2010 results in 20 of the 30 grade and subject areas tested. APS students continue to narrow the gap with State averages.

Dr. Hall wishes Interim Superintendent Davis all success in transitioning APS to its next phase. Great progress has been made since 1999. With continued hard work and dedication, even greater progress can be made in the future.

Like the rest of the public, Dr. Hall awaits the disclosure of the report.

An outline of the report released by Gov. Deal's office includes the following:

• Cheating occurred as early as 2001
• Cheating was caused by a number of factors but primarily by the pressure to meet targets in the data-driven environment
• A culture of fear, intimidation and retaliation existed in APS, which created a conspiracy of silence and deniability with respect to standardized test misconduct
• There were clear and significant warnings of cheating on CRCT as early as December 2005/January 2006, but they were ignored
• The statistics are overwhelming and allow for no conclusion other than widespread cheating in APS

Recently retired APS Superintendent Dr. Beverly Hall says that she "steadfastly denies" that she and her staff, along with the "vast majority of APS teaching and administrative professionals knew or should have known" of widespread cheating during 2009 or any other year.

APS Chair Brenda Muhammad called the findings "devastating" after being briefed on the report early Tuesday.

Members of the Atlanta Public Schools Board, interim APS Superintendent Erroll Davis and Atlanta Mayor Kasim Reed were also briefed on the findings.

Gov. Deal's office is expected to post the full report online some time Tuesday after consulting the State Attorney General to determine if names of educators will have to be removed.

The CRCT report was compiled by two special prosecutors appointed by former Gov. Sonny Perdue to look into allegations of cheating by Atlanta Public Schools on the 2009 CRCT state exams. The appointments were made after a locally-appointed Blue Ribbon Commission failed to determine what happened.

11 Alive interviewed several APS parents who expressed everything from shock to a glimmer of hope.

"It is very disappointing that so many educators felt that this is what they had to do," said Mary Palmer.

"We know that we have done a disservice to our children for years in Atlanta Public Schools and they will be the product of this environment of this community as a result of that failure," said Shawnna Hayes-Tavares.

"It's my hope as a parent and a public school advocate that we can use what we find in this report to move forward and to assure that children won't be cheated ever again," added Julie Davis Salisbury.

Early on, Dr. Beverly Hall, who retired as school superintendent last month, denied that educators were involved in changing student answer sheets to inflate test scores.
But right before she retired, Dr. Hall changed her tune in a video message she e-mailed to employees.

"I expect the investigation to draw some troubling, no alarming, conclusions," Dr. Hall said in late May. "It's become clear that a segment of our staff chose to violate the trust that was placed in them. Let me be clear, there is simply no excuse for unethical behavior and no room in this district for unethical conduct."

"Most educators in this state and most educators in the Atlanta Public Schools are committed, ethical professionals with a passion for educating students," Deal stressed Tuesday. "We owe them a debt of gratitude for their service."

"However, when educators fail to uphold the public trust and students are harmed in the process, there will be consequences," Deal added. "The state has a duty to ensure that students' test results reflect real learning. We cannot allow adult behavior to compromise the very tools which we use statewide to gauge a child's proficiency."

Deal said criminal charges against educators are possible. He plans to send the full report to district attorneys in Fulton, DeKalb and Douglas Counties.

He said the Georgia Professional Standards Commission will also review the findings to determine if there are any violations that could jeopardize teacher certifications.

Viewpoint: SB 1 and raising standards

Over at Prichard, Susan offers some observations on Senate Bill 1 and its opposition to Kentucky's lowered standards.

Since 1990, Kentucky testing has had three scoring systems, each easier than the last. By easier, I mean two things.

First, we set new "cut points" when we changed from 1998 KIRIS to 1999 CATS and when we changed from the 1999-2006 version of CATS to the revised 2007 and 2008 CATS. At the moment that we put the new cut points to work, many more
students were counted as proficient.

Second, we asked for less complex kinds of student performance. From KIRIS to CATS, we dropped performance events and added multiple choice. From "CATS I" to "CATS II" we gave multiple choice more weight than it had before.

Senate Bill 1 demands something different. It calls for P-12 standards to be aligned with college expectations...

Testing Expert Sees ‘Illusions of Progress’ Under NCLB

This from Ed Week:

...Harvard University researcher Daniel M. Koretz has a new book, Measuring Up: What Educational Testing Really Tells Us. Koretz contends that NCLB has prompted widespread teaching to the test, and gaming of the high-stakes testing system producing scores on state standardized tests that are substantially better than students’ mastery of the material.


I have little doubt that Koretz is correct that high-stakes testing prompted some folks to try to game the system. But to illustrate his point, he uses Kentucky's KIRIS assessment which was discontinued by 1998, three years before NCLB. This is a curious choice if one hopes to illustrate a current problem with state systems and NCLB.

Mr. Koretz pointed to research in the 1990s on the state standardized test then used in Kentucky, which was designed to measure similar aspects of proficiency as the National Assessment of Educational Progress, the federally sponsored testing program often called “the nation’s report card.”

Scores on both tests should have moved more or less in lock step, he said. But instead, 4th grade reading scores rose sharply on the Kentucky Instructional Results Information System test, which resembled an NCLB-test prototype, from 1992 to 1994, while sliding slightly on NAEP over the same period.

I'm not sure how it is Koretz determined that KIRIS was in lock step with NAEP. As I recall, the biggest problem with KIRIS was that the damn thing wouldn't hold still - that, and the fact that Kentucky did not even have an underlying curriculum at the time. As Commissioner Thomas Boysen frequently said, we were building the airplane as we flew it. KIRIS was Exhibit A. It changed every year.


Was the NAEP similarly unstable? I don't recall that it was.

Stress Test

This from the Courier-Journal:

More High-Stakes Testing Craziness

This from Gerald Bracey in the Huffington Post.

The Degeneration of American Education

The high-stakes testing mania in general and No Child Left Behind in particular have reduced too much of public education to a system to be games. Some people play the game sincerely and seriously. The teachers and principal in Linda Perlstein's Tested are such players. They have doubts about the value of the state test, but they strive mightily to get their impoverished students over that barrier.

After the test is given in late spring, they start acting like real teachers in a real school -- they take the kids to museums and aquariums and to watch the Blue Angels perform. They make art and write poetry. But only in the short time between the state test and the end of the school year.

Some people play it cynically, doing whatever it takes to get children close to the passing score -- the bubble kids -- off the bubble and into the magic kingdom of "proficient." The "sure things" and "hopeless cases" are ignored. Or emphasizing the increasing passing rates on a required test as students enter their senior year, not taking into account the massive dropouts that have occurred along the way.... that a school can have over 1000 9th-graders, fewer than 300 12th-graders and zero dropouts.

Some play it as if they have lost all sense of proportion and common sense. The Texas Education Agency refused to grant a waiver from the state test for a young woman hospitalized after a serious automobile accident that killed her brother and left her memory impaired. Her school dispatched an assistant principal to administer the test in the hospital. Fortunately, one of the girl's teachers overheard what was up, got to the hospital first and told her to refuse to take it. In Colorado, a father, a teacher himself, sought to opt his daughter out of the state fifth grade test. Fine, said the superintendent, but she won't be promoted to sixth grade.

In Washington, a willing testee who simply couldn't think of how to respond to a writing prompt was harangued by his teacher, then by his principal and then by his mother. Unable to respond, he was forbidden to attend a post-test party at which pancakes were served and the movie Lemony Snicket's A Series of Unfortunate Events shown. He was told he had ruined everything for everyone else at the school and suspended for a week.

And some play it desperately. On March, 27, 2008, the Houston Chronicle reported that a middle school principal told a group of teachers that he would kill them and kill himself if the school's science scores did not improve. He was not, the teachers said, joking. "You don't know how ruthless I can be," he is alleged to have said. The incident is being investigated as a "terroristic threat."

At this point we should be asking HAVE WE GONE COLLECTIVELY MAD?

"Those who can make you believe absurdities can make you commit atrocities," said Voltaire. In a world that contains Clear Skies, Clean Waters, Healthy Forests, an Axis of Evil, Iraqi Freedom, Family Values, Patriot Act, and No Child Left Behind, it is a good reminder for our time.

Teacher burnout? Blame the Parents.

That's the conclusion of a new study published in the journal Anxiety, Stress & Coping.

If you believe that "many of the demands of teaching (including disruptive students, high expectations from school officials and close scrutiny from parents) are universal" you may also be interested in a recent German study of teacher burnout.

Of course, if you are a veteran educator in Kentucky, you already know the impact of management factors (like high-stakes assessment) on the schools. Any study that fails to account for the powerful effects of a high-stakes testing environment is not likely to mean much.

Since about 1995, school reform has created a philosophical shift in the schools from Equality of educational opportunity to equity of student achievement outcomes. This occurred when schools began disaggregating student achievement data - a much more powerful tool for reform than many people initially realized.

Teachers (and principals) are on the front line. They meet the parents daily. They eat what the public feeds them. In that sense, much of the pressure teachers feel under high-stakes assessment is delivered by parents whose expectations are not met by the schools.

But we should all care about this.As the New York Times reported,

...The issue of teacher burnout is important because American schools today are experiencing high levels of teacher turnover as baby boomers retire and new teachers leave the field. According to the most recent Department of education statistics available, about 269,000 of the nation’s 3.2 million public school teachers, or 8.4 percent, quit the field in the 2003-2004 school year. Thirty percent of them retired, and 56 percent said they left to pursue another career or because they were dissatisfied. The National Commission on Teaching and America’s Future has calculated that nearly a third of all new teachers leave the profession after just three years, and that after five years almost half are gone...

The New York Times story concludes,

To be sure, many issues play a role in teacher burnout and turnover. Dwindling school resources, low pay and high expectations for test scores from school districts are just some of the challenges teachers face. But the data from the German study also show that parents can have a big impact on a teacher’s happiness and stress...

Results due this week on academic progress

Every week for the past month, Wilt Elementary School teachers have been asking principal Kimberly Kent the same anxious questions.
"Are the results in yet?" "Do you have them?" "When are they coming in?"

Their questions will be answered this week, when Kentucky's roughly 1,200 public schools get results that will determine whether they have met the academic standards set by the federal No Child Left Behind Act.

For Wilt, and dozens of other schools, there's even more riding on this year's results...

This from the Courier-Journal.

Ed Week Chat With Berliner and Nichols

Highlights of yesterday's Ed Week chat with Berliner and Nichols

on high-stakes testing posdted on School Matters.

Here's a sample:

Question from Walter Carlson, Parent, Fairfax, Virginia:

I agree there are adverse effects of the NCLB but what other choice does society have when after 3 generations the nation's schools have not been able to address the minority achievement gap -- regardless of how much money they've been given. Tests provide a means to identify were problems are and let focus schools on them. Even many educators are admittng that tests do help identify children who need help and schools that are failing. What other choice do we have besides tests?

David Berliner:

I think this is a bogus strength of NCLB and the tests they demand. For at least 50 years I could have told you what schools were not doing well, based on norm referenced low stakes tests, and I could have told you which kids were not doing well. EVERYONE KNEW!!!!!

Nobody had the will to do anything about it--they were not white middle class kids that were the ones who had the trouble, and so these chidren were abandoned.

I see nothing new in NCLB except that the schools are now to fix all of societies ills. The real issue is do we have the desire to invest in the education of poor kids--after school programs, Saturday programs, summer programs, high quality pre schools, small class size for three years, moving high quality teachers to their schools etc...