Kifer on the Problem with an ACT-only Assessment

KSN&C has maintained that what's really wrong with Senate Bill 1 and its suggestion of using the ACT as THE Kentucky test, is that using the ACT doesn't fix the problem. As one part of a comprehensive testing program the ACT may have a place. But alone, it fails.

Sunday, Skip Kifer illustrated the point in the Herald-Leader.

Narrow view of student potential
By Skip Kifer

We old guys remember what the ACT did in its heyday. Along with a student's high school record, ACT scores helped admissions offices decide who had the best chance of succeeding in their institutions. Colleges used ACT scores, along with an array of other information, to place students in courses once they arrived on campus.

Last month, a committee representing the nation's admissions officers released a report about overemphasizing test scores and argued for prudent use of the ACT and the Scholastic Achievement Test in selecting students. The committee suggested that some colleges and universities can enroll students without requiring the tests. This is a recurring theme, not a new one.

What is new is that ACT officials, without having changed the nature of the test, say their scores now tell whether a student "meets expectations" or is "ready" to attend college.

Kentucky's high school juniors are required to take the ACT. Their results, released a couple of weeks ago, were interpreted to mean that too many students failed to "meet expectations" or too few were "ready."

What, we old guys wonder, could these assertions mean?

An ACT score is just that: a score on a test. It tells something about me as a student but means little without a context.

My 18 on the ACT means something very different if I am in the top 10 percent of my high school graduating class rather than just in the top half. My 18 means something different if I have taken solid courses rather than choosing an easy way to a diploma.

My 18 means something different if I were to take the test again. My 18 on the ACT means something different depending on the college or university I attend. Among Kentucky's public universities, for example, my 18 would be a relatively low score in one but a relatively high score in another.

My 18 is like Shaquille O'Neal's free-throw percentage. Being able to shoot barely 50 percent from the free-throw line says something about O'Neal. But, does it mean not "meeting expectations" or not being "ready" for the NBA? Hardly. NBA scouts are clever people who would not let one piece of information become a judgment.

ACT says that a person with an English score of 18 has a 50/50 chance of getting a B or higher in a college English course. We old guys say that is throwing the statistical dice.

My left foot is in boiling water and my right foot is in ice water but "on the average" I feel fine.

Of all students with a score of 18, one-half will get a B or better and one-half a C or worse. In which half is my 18? Is it in the ice or boiling water? ACT cannot answer that question. But if I were in the top 10 percent of my class, it is more likely that I will get the B or better.

We old guys also wonder why so much was made of ACT results and so little reported about Kentucky's Advanced Placement results. As opposed to the ACT, AP scores are a direct measure of whether a student can do college work.

Students take AP courses in a variety of subjects. An AP course is so well defined that what Kentucky students experience is comparable to what is experienced by students throughout the country. Each AP course is taught by a capable teacher and is described in detail with course goals, materials and examinations available for scrutiny.

The examination measures what is learned in the course. High-school students who score 3, 4 or 5 on an AP examination can get placement or credit or both in the college or university they attend.

Recent results show increased numbers of Kentucky's students taking AP courses and scoring higher on them. That is, more Kentucky students are doing acceptable college work in high school. It would make sense, therefore, for an educational community wanting more students to attend college to work to expand AP opportunities.

While it is important to provide opportunities to prepare each student to take AP courses, it is unlikely that all students will take them. Even so, AP courses point in a positive direction for schools. They are a model for structuring courses and tying together testing and instruction.

An algebra course, for example, in Fayette County should be the algebra course in Christian County. More important, two algebra courses in one Fayette County school should be the same algebra course.

AP results show, not surprisingly, that students tend to learn what they are taught. It is important, therefore, to define and teach what it is important to learn. Then one can assess and give credit.

Someone might ask the old guys what this has to do with O'Neal. The answer is clear. If you want to know what kind of basketball player he is, you let him play basketball. You don't look only at his free-throw shooting percentage and decide that he does not meet expectations.

Kifer says EXPLORE and ACT Overstate Claims

Skip Kifer of the Center for Advanced Study of Assessment at Georgetown College released a report this month that calls into question claims made by the ACT program regarding the efficacy of its EPAS system.

Kifer has a history with the ACT... and its a good one. In fact, he has to have been considered a fan. In Mental Measures Yearbook (Kifer, 1985) he lavishly praised test-maker ETS and the technical construction of the ACT. In fact, he found their practices to be worthy of admiration and emulation.

But times change. Since states have begun requiring exams that are specific to state curriculum, ETS has entered the state market and made claims about the efficacy of the EPAS system. It is these claims Kifer examines.

The study was motivated by the introduction of Senate Bill 1 during the 2008 legislative session. The bill included additional reliance on off-the-shelf standardized tests. Since the state assessment already contains such measurements under the EPAS system, a relatively new ACT program, Kifer set out to determine whether the program lived up to its claims.

Kifer looked at the characteristics of EXPLORE and ACT, two components of EPAS taken by 8th, 10th and 11th or 12th grade students, to determine whether or not the claims made for it and them are valid. He also compared ACT predictions to those obtained from other components of the Commonwealth’s assessment.

Kifer concludes that claims for the efficacy of the EPAS are stronger than the findings would suggest.

For example, EXPLORE lacks a representative national comparison group and therefore lacks true national norms. EXPLORE was constructed on the basis of national information, and therefore does not represent the Kentucky curriculum. Kifer also says,

"Although the technical manual suggests that the tests measure 'higher order thinking skills,' there is no dimension for those attributes in the test construction descriptions and no indications of how many questions of that type are on a test. EXPLORE is alleged to be diagnostic but is diagnostic in the limited sense that scores on EXPLORE predict scores on the ACT. Finally, there are validity studies of Kentucky’s assessments and ACT scores that suggest that the Commonwealth’s assessments are comparably effective predictors of performance in higher education."

Kifer concludes that EXPLORE and ACT do not live up to their claims. It also shows that other measures in the Commonwealth’s assessment do about as well and sometimes better than the ACT tests in predicting grade point average in college.

What this means, says Kifer is that,

If a student were certain that he or she was going to attend a public university in Kentucky, it would not be necessary to take the ACT. That student’s CATS score would be sufficiently predictive to take the place of the ACT. A consequence of using CATS scores in a college admissions process might motivate high school students to do their best on the statewide assessments. Finally, not requiring the ACT could save the Commonwealth a substantial amount of money.

Hot off the presses: CASA cites Problems with Senate Bill 1

As KSN&C readers may know that when it comes to assessment, I play favorites.

Over the years, the best, most reliable, and independent sources of information regarding assessment in Kentucky have consistently come from three individuals, Skip Kifer, Ben Oldham and Tom Guskey. I am fortunate to have studied under Ben and Skip at UK. Furthermore, as a principal, I would share my data with Skip annually and he would selflessly sit with me and analyze gap closing and the overall performance of Cassidy students. Tom has built a fine national reputation for scholarly excellence.

If we ever disagree - they're probably right.

If I had a magic wand - they'd redesign and oversee the state's assessment program rather than KDE, but that's another story.

Recently - owing largely to Skip and Tom's disillusionment with some things at UK - they became ripe for recruitment by Vice President Ben Oldham at Georgetown College. Their formation of the Center for the Advanced Study of Assessment is a major coup for the small liberal arts college.

Today they weigh in on Senate Bill 1 in a 19-page analysis posted at the Prichard Committee.

Read the whole thing. But here is their conclusion:

Enacting Senate Bill 1 will change dramatically Kentucky’s assessment and accountability systems. There are four major departures from the existing assessment that we find problematic and should be carefully scrutinized. Careful scrutiny includes identifying changes that have broader implications than might at first be evident.

1. The first major change is moving the focus away from school outcomes to individual academic achievement.

This could substantially reduce the amount of information that is available to schools and districts, information they have used in the past to judge in a particular content area what part of the core content they have done well and what part not so well. The reason for the reduction is that previous assessments produced more than one form for the assessment thereby increasing the breadth and depth of what was sampled. This change also may not have its intended results of better measurements at the individual level. Because the test is expected to be an adequate sample of Kentucky’s core content, produce national norm-referenced results, and provide diagnostic information, the attempt to do all three things may limit how well it does any one of them.

2. The second major change is moving parts of the assessment from the state level accountability portion to the district level where there is no formal accountability.

Moving things to the district level and making the district responsible for parts of the assessment is not necessarily a bad thing to do. Other states have done similar things. But it may not be wise to move just some of the assessment to a new level. An assessment that is part state accountability and part district responsibility may prove to weaken both. We are particularly concerned about the effects of such changes on Kentucky’s long standing commitment to teaching students to write well.

3. The third major change is assessing with only multiple choice items.

Open-ended responses, the writing portfolio, and on-demand writing have been eliminated. Given the nature of the goals, standards, and expectations for Kentucky schools, we do not believe that an assessment that relies exclusively on multiple-choice items can adequately describe those outcomes. Measurements that tap more easily more complex skills and knowledge are both necessary and desirable.

4. The fourth, and perhaps most important, major change is reducing and perhaps eliminating the participation of teachers in the formal assessment.

In the past teachers have worked on content standards, created test questions, evaluated portfolios, and graded open-response items. Although a cadre of teachers became experts in these areas, creating technical expertise was not the main purpose of the involvement. The purpose was to provide opportunities for teachers to see how instruction, standards, and assessmentsshould be intimately tied together. These activities are powerful ways to make teachers both better teachers and better assessors.