For Those Who Would Better Understand the ACT

KSN&C, and I suspect a lot of folks, got a note from Prichard Committee honcho Bob Sexton today regarding the ACT test results recently kicked around in the press.

The media coverage, and occasionally comments from school officials, badly confused what the ACT is and is not and how scores should be used and should not be used.

To bring some scholarly understanding to this misinformation about ACT, Ben Oldham, Distinguished Service Professor at Georgetown College, has written the attached statement.

Here's Ben's article:

September 24, 2008

Adding Understanding to the ACT scores

Ben R. Oldham
Distinguished Service Professor
Georgetown College

The recent release of statewide ACT scores has created a lot of discussion about the quality of Kentucky schools. It has been reported that approximately 43,000 Kentucky juniors earned an average of an 18.3 composite score out of 36 on the ACT. Kentucky is one of just five states that requires the ACT for all high school juniors. It should be noted that the American College Test (ACT) is a highly regarded test developed by a cadre of some of the best measurement professionals in the world and is used by a number of colleges as one selection-for-admission measure.

Extensive research has been conducted that suggests that the ACT is a significant predictor of freshman college grades. The ACT is designed to predict college success. My research suggests that high school grade point average is a similar predictor of college success. Since the ACT is administered to all Kentucky juniors, there is a tendency to over-interpret the results as a measure of the success of Kentucky schools. The successes of Kentucky education reform are inevitably brought into question.

Other evidence tells a different story. More students in Kentucky are taking AP exams and more students are earning college credit through the Advanced Placement program than ever before. These are standards-based exams. The purpose of these tests is to determine, by following a tightly structured curriculum, if students earn high enough scores to earn college credit while still in high school. Teachers know precisely what should be taught and through their excellent instruction more high school seniors earned college credit.

The Kentucky Core Content Test (KCCT) is administered throughout the grades of Kentucky’s public schools. Like AP tests, the KCCT assessments are standards-based exams. Teachers in Kentucky’s schools teach from a core content that defines what Kentucky students should know and be able to do as they progress through school. Like the AP test, the KCCT is designed to precisely measure how well the students have mastered the defined curriculum. The categories of novice, apprentice and distinguished are used to define the achievement of students. Its purpose is to monitor the growth of schools toward a Commonwealth goal of the average student achieving at the proficient level by 2014.

It is desirable for large numbers of students to achieve at the highest levels of achievement on both the AP and KCCT tests. Having small and reducing numbers of students at the lowest levels of achievement is also desirable. Here is where the difference with the ACT and standards-based tests lies. The ACT is a norm-based test. It is not designed to determine what students know and are able to do like the AP test or KCCT test. A norm-based test compares a student’s performance on a bank of test questions with students in a comparison group; in this case a national but not nationally representative comparison group since its purpose is for the college-bound. By design, the ACT spreads student scores to assist colleges and universities in making admission and scholarship decisions. When the ACT was developed, the average score was set at 20 regardless of the academic achievement of those in the norm group. If it were the case that everyone in the national comparison group scored at a high level, the mean score would be 20. If nearly all scored at a low level on the ACT, the mean would be 20. The purpose of the ACT is to assist colleges and
universities in making admission decisions. By design it separates students into a range of scores from the 1st percentile to the 99th percentile regardless of the pure academic achievement.

Because it is administered to students across the country, the ACT is designed to be insensitive to curriculum to not give an advantage to any particular curriculum. This is another major difference. Both the AP and the KCCT are built around a tightly defined curriculum. Because the ACT is insensitive to school curricula many employ the test-taking strategies to artificially inflate test scores. It should not be quick and easy process to improve test scores because it does not reflect true improvement. However, given a defined curriculum, public school teachers have done and will continue to do an exemplary job educating students toward a common goal. Are there improvement strategies that can be employed? Absolutely, but the ACT does not contribute to these strategies because the ACT must, by design, separate students to assist colleges in selection decisions.

Teachers, parents, principals, superintendents, board of education members, and most importantly students must not overlook this purpose. While it is important, schools should not be evaluated using a tool that is insensitive to the core content and is designed to differentiate between the higher-achieving college-bound.

If the purpose of Kentucky public schools is to prepare all elementary and secondary school students for college, then the core content followed by schools needs to be adjusted with significant input from college professors to include things like the thoughtful analysis of data and ideas, explaining and demonstrating math solutions and a solid foundation in the college general education curriculum. Regardless, the evaluation of the achievement of the core content must be measured by a test that determines the success in achieving that curriculum rather than a norm-based instrument, like the ACT, that merely compares a student’s performance with college-bound students nation-wide.