Kifer says EXPLORE and ACT Overstate Claims

Skip Kifer of the Center for Advanced Study of Assessment at Georgetown College released a report this month that calls into question claims made by the ACT program regarding the efficacy of its EPAS system.

Kifer has a history with the ACT... and its a good one. In fact, he has to have been considered a fan. In Mental Measures Yearbook (Kifer, 1985) he lavishly praised test-maker ETS and the technical construction of the ACT. In fact, he found their practices to be worthy of admiration and emulation.

But times change. Since states have begun requiring exams that are specific to state curriculum, ETS has entered the state market and made claims about the efficacy of the EPAS system. It is these claims Kifer examines.

The study was motivated by the introduction of Senate Bill 1 during the 2008 legislative session. The bill included additional reliance on off-the-shelf standardized tests. Since the state assessment already contains such measurements under the EPAS system, a relatively new ACT program, Kifer set out to determine whether the program lived up to its claims.

Kifer looked at the characteristics of EXPLORE and ACT, two components of EPAS taken by 8th, 10th and 11th or 12th grade students, to determine whether or not the claims made for it and them are valid. He also compared ACT predictions to those obtained from other components of the Commonwealth’s assessment.

Kifer concludes that claims for the efficacy of the EPAS are stronger than the findings would suggest.

For example, EXPLORE lacks a representative national comparison group and therefore lacks true national norms. EXPLORE was constructed on the basis of national information, and therefore does not represent the Kentucky curriculum. Kifer also says,

"Although the technical manual suggests that the tests measure 'higher order thinking skills,' there is no dimension for those attributes in the test construction descriptions and no indications of how many questions of that type are on a test. EXPLORE is alleged to be diagnostic but is diagnostic in the limited sense that scores on EXPLORE predict scores on the ACT. Finally, there are validity studies of Kentucky’s assessments and ACT scores that suggest that the Commonwealth’s assessments are comparably effective predictors of performance in higher education."

Kifer concludes that EXPLORE and ACT do not live up to their claims. It also shows that other measures in the Commonwealth’s assessment do about as well and sometimes better than the ACT tests in predicting grade point average in college.

What this means, says Kifer is that,

If a student were certain that he or she was going to attend a public university in Kentucky, it would not be necessary to take the ACT. That student’s CATS score would be sufficiently predictive to take the place of the ACT. A consequence of using CATS scores in a college admissions process might motivate high school students to do their best on the statewide assessments. Finally, not requiring the ACT could save the Commonwealth a substantial amount of money.

Hot off the presses: CASA cites Problems with Senate Bill 1

As KSN&C readers may know that when it comes to assessment, I play favorites.

Over the years, the best, most reliable, and independent sources of information regarding assessment in Kentucky have consistently come from three individuals, Skip Kifer, Ben Oldham and Tom Guskey. I am fortunate to have studied under Ben and Skip at UK. Furthermore, as a principal, I would share my data with Skip annually and he would selflessly sit with me and analyze gap closing and the overall performance of Cassidy students. Tom has built a fine national reputation for scholarly excellence.

If we ever disagree - they're probably right.

If I had a magic wand - they'd redesign and oversee the state's assessment program rather than KDE, but that's another story.

Recently - owing largely to Skip and Tom's disillusionment with some things at UK - they became ripe for recruitment by Vice President Ben Oldham at Georgetown College. Their formation of the Center for the Advanced Study of Assessment is a major coup for the small liberal arts college.

Today they weigh in on Senate Bill 1 in a 19-page analysis posted at the Prichard Committee.

Read the whole thing. But here is their conclusion:

Enacting Senate Bill 1 will change dramatically Kentucky’s assessment and accountability systems. There are four major departures from the existing assessment that we find problematic and should be carefully scrutinized. Careful scrutiny includes identifying changes that have broader implications than might at first be evident.

1. The first major change is moving the focus away from school outcomes to individual academic achievement.

This could substantially reduce the amount of information that is available to schools and districts, information they have used in the past to judge in a particular content area what part of the core content they have done well and what part not so well. The reason for the reduction is that previous assessments produced more than one form for the assessment thereby increasing the breadth and depth of what was sampled. This change also may not have its intended results of better measurements at the individual level. Because the test is expected to be an adequate sample of Kentucky’s core content, produce national norm-referenced results, and provide diagnostic information, the attempt to do all three things may limit how well it does any one of them.

2. The second major change is moving parts of the assessment from the state level accountability portion to the district level where there is no formal accountability.

Moving things to the district level and making the district responsible for parts of the assessment is not necessarily a bad thing to do. Other states have done similar things. But it may not be wise to move just some of the assessment to a new level. An assessment that is part state accountability and part district responsibility may prove to weaken both. We are particularly concerned about the effects of such changes on Kentucky’s long standing commitment to teaching students to write well.

3. The third major change is assessing with only multiple choice items.

Open-ended responses, the writing portfolio, and on-demand writing have been eliminated. Given the nature of the goals, standards, and expectations for Kentucky schools, we do not believe that an assessment that relies exclusively on multiple-choice items can adequately describe those outcomes. Measurements that tap more easily more complex skills and knowledge are both necessary and desirable.

4. The fourth, and perhaps most important, major change is reducing and perhaps eliminating the participation of teachers in the formal assessment.

In the past teachers have worked on content standards, created test questions, evaluated portfolios, and graded open-response items. Although a cadre of teachers became experts in these areas, creating technical expertise was not the main purpose of the involvement. The purpose was to provide opportunities for teachers to see how instruction, standards, and assessmentsshould be intimately tied together. These activities are powerful ways to make teachers both better teachers and better assessors.