Kifer says EXPLORE and ACT Overstate Claims

Skip Kifer of the Center for Advanced Study of Assessment at Georgetown College released a report this month that calls into question claims made by the ACT program regarding the efficacy of its EPAS system.

Kifer has a history with the ACT... and its a good one. In fact, he has to have been considered a fan. In Mental Measures Yearbook (Kifer, 1985) he lavishly praised test-maker ETS and the technical construction of the ACT. In fact, he found their practices to be worthy of admiration and emulation.

But times change. Since states have begun requiring exams that are specific to state curriculum, ETS has entered the state market and made claims about the efficacy of the EPAS system. It is these claims Kifer examines.

The study was motivated by the introduction of Senate Bill 1 during the 2008 legislative session. The bill included additional reliance on off-the-shelf standardized tests. Since the state assessment already contains such measurements under the EPAS system, a relatively new ACT program, Kifer set out to determine whether the program lived up to its claims.

Kifer looked at the characteristics of EXPLORE and ACT, two components of EPAS taken by 8th, 10th and 11th or 12th grade students, to determine whether or not the claims made for it and them are valid. He also compared ACT predictions to those obtained from other components of the Commonwealth’s assessment.

Kifer concludes that claims for the efficacy of the EPAS are stronger than the findings would suggest.

For example, EXPLORE lacks a representative national comparison group and therefore lacks true national norms. EXPLORE was constructed on the basis of national information, and therefore does not represent the Kentucky curriculum. Kifer also says,

"Although the technical manual suggests that the tests measure 'higher order thinking skills,' there is no dimension for those attributes in the test construction descriptions and no indications of how many questions of that type are on a test. EXPLORE is alleged to be diagnostic but is diagnostic in the limited sense that scores on EXPLORE predict scores on the ACT. Finally, there are validity studies of Kentucky’s assessments and ACT scores that suggest that the Commonwealth’s assessments are comparably effective predictors of performance in higher education."

Kifer concludes that EXPLORE and ACT do not live up to their claims. It also shows that other measures in the Commonwealth’s assessment do about as well and sometimes better than the ACT tests in predicting grade point average in college.

What this means, says Kifer is that,

If a student were certain that he or she was going to attend a public university in Kentucky, it would not be necessary to take the ACT. That student’s CATS score would be sufficiently predictive to take the place of the ACT. A consequence of using CATS scores in a college admissions process might motivate high school students to do their best on the statewide assessments. Finally, not requiring the ACT could save the Commonwealth a substantial amount of money.