Is K-PREP the Enemy of Good Schools?

It appears that the Commonwealth's commitment to a school reform 

based on fairness and equality of opportunity has vanished.

---Skip Kifer

On the 11th, the editorial board at the Herald-Leader weighed in on the new K-PREP tests. They admit to being flummoxed by much of the K-PREP system. Join the crowd. KSN&C has heard from a number of educators who aren't exactly sure how to comprehend the test and more than a few wonder if they should trust the results.

Some fear that the primacy given to accountability has caused the state to stretch the technical limits of assessment far beyond good design and has distracted the state from providing sufficient instructional improvement support for teachers as a result.

But when Kentucky testing expert Skip Kifer looked at the data produced by the old KIRIS test and compared it to the K-PREP he had a hard time finding any news.

I was taken by what appeared to be about the same rank order for schools as was found in 1990. Schools in wealthy communities do well; those in impoverished areas do not. And people still blame the schools for not creating economic equality. 

This already complex K-PREP system will only become more inaccessible when Kentucky adds Program Reviews to the top of the pile next year and new cut scores are established. How long will it take such a complex system to crumble under its own weight?

In back-to-back posts we present the Herald-Leader Editorial Board's take on K-PREP followed by Skip Kifer's response.

Putting schools to the test 

 

Much to like in new system 

but shortcomings must be addressed

This from H-L:

We hope Kentucky educators can mine loads of useful data from the new school assessments because the average parent or taxpayer who tries to drill down into the statistical murk will come away frustrated.

Also, Education Commissioner Terry Holliday and the state school board must quickly address two big concerns:

■ How can the new system drive schools to improve when their annual goals are set so low?
■ Why, when Kentucky graduates must compete globally, are we basing school performance standards not on the best schools in the world but the highest-scoring schools in Kentucky?

Before explaining our concerns, we must applaud the many good changes Kentucky has made since 2009. As the first state to adopt national standards in reading and math, Kentucky is already giving students richer, deeper, more rigorous content in the classroom.

Where the wheels start falling off is the new plan for grading schools. Again there's much to like, such as finally more accurately measuring dropout rates.

What's questionable is whether the new system will drive school improvement fast enough, especially in schools where only a few students are proficient in reading and math.

Even the lowest-performing schools, those scoring in the 20s on a 100-point scale, must improve by just 1 point by next year.

That's a snail's pace, especially if your child is stuck in an under-performing school.

As far as we can tell — and we admit, we're flummoxed by much of K-PREP — the pace of required improvement will pick up only a very little in the future, consigning too many kids to bad schools for too long.

Also, for schools that fall short, there are no consequences and state assistance for only a few.

In fairness, Holliday and the Department of Education had to develop a new accountability system on the cheap because the legislature threw out the old without providing any money to build the new. The state outsourced creation of the test that students take to Pearson, a multinational, for-profit company.

The process seemed to break down when it came to grading school performance. A decision was made to grade schools, not on any agreed upon criteria, but based on the scores of the top 30 percent in Kentucky.

School-level accountability was one of the things taxpayers got in 1990 in return for a penny sales tax increase to pay for education reform and equalize funding for poor districts.

You can't expect the public to pay more to expand early childhood education and make other improvements, as Kentucky must to compete, without also measuring and publishing the results of that spending in a way that people can understand and trust.

Kentucky's new system is still very much a work in progress, which leads to our final question for Holliday, the state school board and lawmakers: Will a state that pioneered education accountability settle for a system that is so confusing, user-hostile and lax as to render it irrelevant? Please, say no.

Is K-PREP Really a Better Mouse Trap?

This from Skip Kifer:


[Last] Sunday's editorial in the Herald-Leader should be a catalyst for discussion of the Commonwealth's new assessments. The board's broad characterization of the new assessment regime is right on. I googled Rube Goldberg to see if that might be an appropriate description. It's not. His complex machines apparently would do the simple things they addressed.

I think, however, improving schools and holding schools accountable get conflated in the editorial.

In 1994 I wrote:

Kentucky's assessment system is large, complex, and ambitious. It aims to do two, perhaps incompatible, things: first, to provide a statewide, school-level accountability system; and second, to produce dramatic changes in curriculum and instruction in public schools. In terms of the latter, it is based on the premise that exemplary assessment procedures will produce optimal instruction. Teaching to the assessments, it is believed, will not only produce increasingly higher test scores but will also drive desirable instructional practices.

So far, it appears that more emphasis has been placed on the accountability portion of the assessment system than on either of the components of continuous assessment. Because increasingly higher proportions of the assessments will be either performance tasks or portfolios, it is crucial that teachers be trained to conduct these and to use them routinely in their classrooms.

Then, I thought for assessment activities to have a positive effect on classroom instruction it was necessary to have both exemplary assessment tasks and good information and training for teachers. We tried to do that but failed. And the failure is attributable to the emphasis on school accountability. When the performance assessments were thrown out, they were thrown out because they were not sufficiently reliable and valid for the accountability piece. The portfolio assessments, so they could be used in the accountability system, were so burdened by rules of engagement that they no longer could serve the instructional piece.

Teachers then were involved in item writing, scoring of short response and longer response questions, evaluating portfolios, and other activities based on the premise that understanding the rationales and procedures for the assessment would translate into more effective instruction: Being able to distinguish good answers from bad ones and good essays from mediocre ones was tied to better instruction. No more.

Back then I was less than enthusiastic about the accountability piece. I now believe accountability system based on test scores is the enemy of good schools. As Michael Scriven says: "Side effects are often the main event."1

The editorial mentions performance standards. Early on, when NAEP introduced achievement levels (they are still, after about 30 years, considered experimental because no one will endorse them), such things (performance standards, proficiency levels, cut-scores) were labeled arbitrary but not capricious. Perhaps the new Kentucky ones are both.

In a related matter, the editorial raises issues about norms, wondering why not use international comparisons. There is nothing intrinsically wrong with normative score interpretations. But, it matters little what the norm group is. What is important is what talents are developed in our schools.

The editorial praises the state for being the first to adopt national standards, the so-called common core. It is important to mention, however, that the present assessments are not those funded by the US Department of Education to assess the common core. Those two assessments have not yet been completely developed, tried out, or implemented. How Kentucky's present assessments are related to the common core is an interesting question.

In other matters:

I was taken by what appeared to be about the same rank order for schools as was found in 1990. Schools in wealthy communities do well; those in impoverished areas do not. And people still blame the schools for not creating economic equality. I am willing to bet there is as much or more inequality of funding now than in 1990. In fact, I have some data that suggests as much. I bet such things as extended school services and family resource centers are not so well-funded as they were initially. It appears that the Commonwealth's commitment to a school reform based on fairness and equality of opportunity has vanished.

KDE officials in the first utterance say that one cannot compare this assessment's results to the previous ones. The next utterance is that the scores are going to be much lower. Hum...

High schools must feel better. They have improved greatly according to the scores.

It took England, in 1862, 33 years to get rid of Payment by Results.2 Kentucky has only about 10 years left... 

Read more here: http://www.kentucky.com/2012/11/11/2403587/putting-schools-to-the-test-much.html#storylink=cpy

1.Michael Scriven is a British-born polymath and academic, best known for his contributions to the theory and practice of evaluation.

2. "Payment by results," a rigid method of accountability associated with English and Welsh elementary education during the second half of the nineteenth century, was a system whereby a school's governmental grant depended for the most part on how well pupils answered in the annual examination conducted by Her Majesty's Inspectors. See Rapple, B.A. (1994). Payment by results: An example of assessment in elementary education from nineteenth century Britain. Educational Policy Analysis Archives. 2. 1-21.

1

Read more here: http://www.kentucky.com/2012/11/11/2403587/putting-schools-to-the-test-much.html#storylink=cpy