Standards board takes action against Perry county educators in ACT investigation

This from the Hazard Herald:

The Education Professional Standards Board this week suspended educational certificates for two employees of the Perry County School District in response to an investigation that turned up allegedly compromised ACT tests in 2009 and 2010.

The ACT organization, which provides college entrance exams to high school students in Kentucky and other states, confirmed to the Herald in October 2010 that officials had cancelled test scores for several students at Perry County Central, and later at Buckhorn High School, after an investigation concluded that answers on some of the tests had been altered and did not reflect “the students’ own independent work.”

ACT forwarded the results of their investigation to the Kentucky Department of Education and the Education Professional Standards Board (EPSB), the latter of which opened cases on five employees of the district. Two of those cases were closed with a suspension of certification on Monday, January 9.

According to documents obtained by the Herald this week through an open records request, Jonathan Jett, who at the time served as the district’s assessment coordinator, was “responsible for test security in the district,” and “should have been aware that ACT scores for certain student groups appeared to be inconsistent with their abilities.”

An agreed order was issued during the EPSB’s meeting on Monday, noting punitive measures against Jett and his certification in Kentucky.

“Although Jett maintains his innocence with regard to the specific allegation,” the order reads, “he acknowledges that the evidence regarding the reported charges is such that, if presented at a hearing of this matter, could result in a finding that he is in violation of KRS 161-120 and 16 KAR 1:020, the Professional Code of Ethics for Kentucky School Certified Personnel.”

Two of the three educational certificates Jett holds were suspended as a part of the order. One listed as a “Professional Certificate For Instructional Leadership–School Superintendent” was suspended for 12 months beginning July 1, 2011 and ending July 1, 2012, while a “Professional Certificate For Director Of Special Education” was suspended for 18 months beginning July 1, 2011.

According to Alicia Sneed, director of legal services for the EPSB, Jett additionally holds a teaching certificate which will not be affected “since the allegations in the case were in reference to his positions as District Assessment Coordinator and Director of Special Education.”

Additional probationary conditions were also attached to the order, including one that bars Jett from supervising the administration of tests required by state statute or by the Kentucky Department of Education.

A second case was closed on Monday against Rebecca Dobson, a certified teacher at Perry Central who, according to the agreed order in her case, was serving as the testing accommodations coordinator at the time ACT alleged the tests were altered. The order also noted that her fingerprints were allegedly found on the answer sheets, according to the original complaint made by ACT.

Dobson also maintained her innocence, according to the documentation, but agreed to the order which suspends her teaching certification for a period of six months beginning September 3, 2011 and ending on February 3, 2012, after which her certificate will be reinstated under a probationary period of five years. Conditions of that probation include that Dobson provide proof that she completed 12 hours of ethics training and that she not be allowed to participate in any testing mandated by state law or by the Kentucky Department of Education.


Perry County Superintendent John Paul Amis noted on Wednesday that Jett’s suspension will not affect his current job as the district’s maintenance and transportation director, and that Dobson is currently on an unrelated leave and isn’t scheduled to return until next month.

He added that the suspensions levied by the EPSB were unfortunate because it appeared that both Jett and Dobson were punished simply by virtue of their supervisory positions, and that to his knowledge there has never been a specific allegation or evidence presented that either had anything to do with physically altering the test’s answer sheets.

“It’s my understanding the only reason the standards board has taken any action against any individuals is due to negligence, because of the positions they held, and not willful negligence, just negligence, that these people never willingly, knowingly participated in anything improper, but due to the positions that they held some disciplinary action was taken against them,” Amis said.

In Jett’s case, Amis noted, he was never in contact with the answer sheets at any time during their administration, and in Dobson’s case, it would be natural for someone administering the test to have their fingerprints on the booklets.

“I’m sure if they checked, there would be numerous fingerprints on them, because somebody has to hand them out and somebody has to take them up, and somebody has to inventory them and seal them up,” he said. “That wouldn’t be evidence of guilt on anybody’s part.”

Amis said he has reviewed the evidence contained in the ACT report several times, and while it does seem that some manipulation of the tests may have taken place, it has been frustrating that no one has been able to pinpoint any specific individual as the person who physically altered the students’ answers.

“When you look at the evidence of some of these answer sheets … the evidence is pretty compelling that some of those could have been manipulated,” he said. “But again, a lot of people don’t realize how in-depth ACT’s investigation was.”

That investigation, he noted, included forensic elements such as fingerprint and handwriting analysis. And despite that inquiry and another completed by district officials on the local level, “they could still never determine an individual who actually went in and changed any of the answers, (or) manipulated those answer sheets in any way.”

And that has been a point of frustration, he added. “We would have dealt with an individual severely if we could ever have determined exactly who it was.”

“It’s unfortunate,” Amis continued, “that you’ve got people that their names are now going to be made public and may be some action taken against them with their certifications when they’ve never been accused of actually doing anything improper.”

Three other cases are currently pending, the details of which are not being released. Alicia Sneed with the EPSB explained that those files will also become public record once the cases are closed. She did not note a time frame as to when that may happen.

Superintendent Amis also declined to release any specifics in those remaining cases, but said he understands that like the two that were closed this week, those individuals were named because of their positions and not because of any specific allegations or evidence against them.

In the original ACT complaint, tests were also alleged to have been compromised at the county’s other public high school at Buckhorn, though Sneed declined to comment if there are any plans to open separate investigations in regard to those allegations.

Amis did say, however, that he received a letter from ACT several months ago that the agency was not pursuing anything with Buckhorn, and that he doesn’t expect any further investigation into those allegations.

Holliday on the ACT Announcement

This from the Commish at Dr H's Blog:

Operation Preparation

This week, Kentucky was host to the annual release of ACT results. The annual data release is usually held in Washington, D.C. This year, ACT, Inc. wanted to highlight the national focus on college and career readiness.

When ACT staff called to ask me if they could coordinate the national release with our state release, they said Kentucky was one of the leading states in implementing the college and career readiness agenda in the nation. We chose the Jessamine County Career and Technical Center to highlight the work of Superintendent Lu Young and her community in implementing the Commonwealth Commitment for College and Career Readiness. While we did have to choose one location for the announcement, the location represented the tremendous work that all schools and districts in the Commonwealth are doing to implement the Commonwealth Commitment.

ACT President Jon Erickson highlighted the areas where the nation and Kentucky have improved and also offered several suggestions for improvement at the state and national levels. We were informed that, of the four states who have made significant progress on ACT results, Kentucky ranks second...

Kentucky juniors improve slightly on ACT, but most still aren't college-ready

This from the Herald-Leader:

Scores on the ACT test taken by juniors in Kentucky's public high schools in the spring improved slightly over last year in most subject areas, according to results released Wednesday by the state Department of Education.

The composite score for Kentucky juniors this year was 18.8, up from 18.5 in 2010.

Average scores in English rose from 17.8 last year to 18 this year. Math went up from 18.3 to 18.5; reading from 18.9 to 19, and science from 18.7 to 19.

Students who achieve so-called "benchmark" scores or higher on the ACT are considered ready for college-level courses. Those benchmark scores are 18 in English, 22 in math, 21 in reading and 24 in science.

According to ACT officials, students who reach benchmark scores have a 50 percent chance of getting a grade of B or higher, or a 75 percent chance of getting a C or higher, in corresponding first-year college credit courses.

Test data released Wednesday show that the percentages of Kentucky juniors ready for college work have increased fairly steadily since Kentucky started requiring juniors to take the ACT in 2008.

Search ACT data at H-L.
In a story released yesterday in H-L:

The percentage of Kentucky high school graduates reaching benchmark scores on the ACT test improved in some subjects this year, but educators still have much work to do, results being released Wednesday show.

For example, more than 35 percent of Kentuckians who graduated from high school this spring failed to achieve any benchmark score on the ACT. The benchmark scores indicate likely success in college.

The Kentucky numbers are included in the national 2011 Condition of College and Career Readiness Report, which ACT Inc. [released] Wednesday morning at the Jessamine Career and Technology Center in Nicholasville...

In an apparent effort to be fair and balanced the story quoted a politician from the Bluegrass Institute who made no effort toward either.

Richard Innes, an analyst with the free-market Bluegrass Institute for Public Policy Solutions, contended that scores from Louisiana, another 100 percent testing state, show the potential of charter schools. Louisiana opened large numbers of charters after Hurricane Katrina in 2005.

Innes noted that Louisiana's 2011 ACT composite score was 20.2, compared with Kentucky's 19.6. Louisiana's scores for white and African-American students also were significantly higher than Kentucky's, he said.

"There definitely are indications here that a well-run charter program could help us a lot," Innes said.

The fact that Innes unabashedly suggested a causal relationship between charters and the ACT is not surprising - not as surprising as the fact that H-L continualy returns to him and other politicians at BIPPS as sources for education analysis in news stories. Their stuff belongs on the opinion pages.

For example, Innes has cited Tennessee's value-added system as a shining example of what Kentucky should aspire to. Yet, "Kentucky, with an overall composite ACT score of 19.6, edged ahead of Tennessee, whose composite was 19.5." Is that proof enough for Innes that value-added systems don't work?

Where is the Bluegrass Institute's press release abandoning their support for such systems?

National ACT Data to be Released at Press Event Held at the Jessamine Career and Technology Center

On August 17, ACT, Inc. will release the 2011 Condition of College and Career Readiness Report and data from the nationwide ACT administration for 2011 graduating seniors at the Jessamine Career and Technology Center in Nicholasville. This is the first time ACT, Inc. has held the release in a state and Kentucky was chosen due to its focus on college and career readiness.          

The event will begin at 10:00 a.m. ET and features Jessamine County Superintendent Lu Young, Kentucky Education Commissioner Terry Holliday, ACT Education Division Interim President Jon Erickson, Council on Postsecondary Education President Robert King, State Representative Carl Rollins and State Senator Ken Winters.

Data to be released include information about graduating seniors meeting ACT benchmarks and results of the 2011 ACT administration to those students.

On August 17, the Kentucky Department of Education also will release data from the 2011 administration of the ACT to the state’s public school juniors.

The public is welcome to attend and the event will be webcast live.  You can access it by clicking here.  

ACT Press Event

Wednesday, August 17, 2011 | 10:00 to 11:00 am

Jessamine Career and Technology Center | Nicholasville, KY

10:00 to 10:10 am     Welcome, Introductions, Context Setting
Lu Young, Superintendent, Jessamine County Schools

10:10 to 10:30 am     The 2011 ACT Condition Report
Jon Erickson, Interim President, Education Division, ACT

10:30 to 10:45 am      The Kentucky Agenda
Terry Holliday, Commissioner of Education, Kentucky

10:45 to 10:55 am     Reactions from Kentucky State Leaders

• Robert King, President, Kentucky Council on Postsecondary Education
• Representative Carl Rollins, Co-Chair of the Kentucky General Assembly’s Interim Joint Committee on Education
• Senator Ken Winters, Co-Chair of the Kentucky General Assembly’s Interim Joint Committee on Education

10:55 am                     Facilitated Questions
                                   Lu Young

SOURCE: KDE Press release and Commissioner's Fast Five

New Study Questions Validity of Two Parts of ACT

This from Ed Week:

The validity of the ACT in predicting college success has come under scrutiny in a new paper out by the National Bureau of Economic Research.

The study, Improving College Performance and Retention the Easy Way: Unpacking the ACT Exam, by Eric P. Bettinger, Brent J. Evans, and Devin G. Pope, suggests that two of the four sub tests of the ACT, English and mathematics, are highly predictive of positive college outcomes, while the other two, science and reading, provide little or no additional predictive power.

Colleges rely on the ACT exam in their admission decisions to increase their ability to differentiate between students likely to succeed and those that have a high risk of under-performing and dropping out. We show that two of the four sub tests of the ACT, English and Mathematics, are highly predictive of positive college outcomes while the other two subtests, Science and Reading, provide little or no additional predictive power. This result is robust across various samples, specifications, and outcome measures. We demonstrate that focusing solely on the English and Mathematics test scores greatly enhances the predictive validity of the ACT exam.

Officials at the ACT, based in Iowa City, Iowa, maintain the test is supported by decades of research.

King Clarifies Stance on ACT and College Admissions

Council on Postsecondary Education President Bob King recently offered his opinions on college admissions saying,

Historically, parents are often directed to focus on graduation rates and average GPAs as evidence of how their high school is performing. Our reports allow parents and educators to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam — now required of all Kentucky students.

That drew a response from KSN&C's Skip Kifer.

Bob King, president of Kentucky's Council on Postsecondary Education applies the council's arbitrary standard of using a single test score to determine whether a student is ready for regular course work in Kentucky's public universities.

He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above-average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's readiness for higher education.

Today, King clairifed his stance in the Herald-Leader.

...Please allow me to clarify that Kentucky's colleges and universities do not rely exclusively on the ACT to make college admission or placement judgments, nor does the Council on Postsecondary Education encourage such determinations.

A student's entire record, including GPA, extracurricular activities, and other placement exams form a portfolio that allows campuses to make informed decisions on admission and placement.

The ACT serves as an important element in this consideration, but more importantly, it serves as an alarm bell in the student's secondary experience about preparation for life after high school.

This might have been a good place to stop. It acknowledges Kifer's concerns and clarifies King's stance. But King then makes allusions to "certain thresholds" in the ACT which serve to warn us if a student is not on track.

It warns of the need to take a deeper look at a student's college readiness if the scores fall below certain thresholds, but it is not the sole determinant when placing students in developmental courses... Far from arbitrary cutoff scores, there is a great deal of data from tens of millions of ACT score results upon which policy makers in Kentucky rely to set the scores used to indicate college readiness in key entry-level courses.

If the thresholds King has in mind are, in fact, a reference to ACT's benchmarks, which are inappropriately modeled, one wonders if King's effort to lay the issue to rest might draw yet another response from Kifer.

We'll see.

Famous Last Words

Amis says ACT inquiry in Perry a ‘dead issue’

This from the Hazard Herald:

An investigation into compromised test scores in Perry County is no longer an issue and districts officials have moved on, Superintendent John Paul Amis told board members during a meeting of the Perry County Board of Education last week.

In 2010, ACT, based in Iowa, launched an investigation into tests given at Perry County Central and Buckhorn High School, noting that information found during the inquiry led investigators to believe that the resulting tests were not solely the work of the students. Several scores were canceled and students were given the opportunity to retake the exam.

ACT took little action following their investigation, but did require the district to submit remediation reports and strengthen testing protocols.

Superintendent Amis took a moment during last week’s meeting to chide local media outlets for their recent coverage of ACT’s investigation, saying the now concluded inquiry is a “dead issue.”

Details of the ACT’s findings were included in two news stories in Hazard last week, first from the Hazard Herald and then from WYMT-TV after the Herald made an an open records request to the Kentucky Department of Education seeking documents related to the investigation.

According to those documents, ACT investigators found among other things high erasure counts indicating altered test responses and were told by at least one student that adult proctors had given answers to students during the test.

Amis said no students have told local officials of any impropriety, adding that there was no reason to be reporting on the story as he considers it over with. He said the investigation concluded two months ago, and there were no Perry County Schools employees implicated in the final report. He also noted that both Perry Central and Buckhorn will continue giving the ACT locally.

“It’s over,” he said. “It’s a dead issue. The only people that are keeping that issue alive is the media.”

Amis said he has never had a parent call him upset with the district about the investigation. He said there has been some anger toward ACT over their handling of the investigation, and now with the media.

“I had people today tell me that they’ll never buy another Hazard Herald or watch WYMT again because you just won’t let this issue go,” he continued. “The media and some of our school rivals are the only people that’s even mentioning this thing.”

Amis also criticized ACT for not sharing information with the district about who may have been responsible for manipulating the tests, and again noted that the final investigative report made no note of specific individuals either.

Board member Jerry Stacy said he has received calls from people about the investigation, and finds some of the allegations “troubling.”

“I was going through this today, and I couldn’t believe some of these allegations,” Stacy said during Thursday’s meeting. “I think these allegations are damning.”

Stacy said he was curious as to why the information was forwarded to other state agencies such as the attorney general’s office.

“That’s a dead issue,” Amis replied, saying that he thought ACT had a hidden agenda and wanted to hurt the Perry County District’s image.

“All ACT is trying to do ... is just to give the district a black eye. That’s all they’ve done,” he continued.

“We have done everything that they have asked us to do, and we’re going to continue to give the ACT at both of our high schools, and it’s over as far as I’m concerned,” he added.

Amis said despite charges of testing manipulation, the majority of the students scored the same or higher on the test during the make-up exam.

“We had 116 students that retook that ACT, and over 60 percent of them made the same score or a higher score than what they made the first time, but they won’t report that,” Amis said. "Nobody reports that.”

The final report issued by ACT on November 23 notes that 59 students whose scores for the 2010 ACT were canceled actually saw an average decrease in their new score by one point. But Amis said ACT cherry picked those students in an effort to ensure a negative reflection upon the school district. He said ACT’s focus on only 59 scores didn’t tell the whole story.

“They could take the 59 and make it appear like the scores went down,” he said.

“You actually think they’ve (ACT) got some kind of hidden agenda?” Stacy later asked.

“Obviously,” Amis replied.

Scott Gomer, media relations director with ACT, said the organization had no hidden agenda in its investigation, but did note that ACT’s part in the matter has concluded and offered no further comment.

According to an email from Lisa Gross with the Kentucky Department of Education (KDE) in December, the matter was referred to the Education Professional Standards Board (EPSB) for review, as well as the attorney general’s office and the U.S. Dept. of Education’s Office of Inspector General.

Gross said KDE has had no word that either of those agencies have finalized their own investigations.

“Our legal counsel remains in contact with all of the agencies mentioned, but we’re not aware that any investigations outside of ACT’s have been completed,” she wrote in an email on Monday.

The Education Professional Standards Board did receive a referral from KDE, noted Alicia Sneed, director of legal services for EPSB, but thus far the agency has taken no action.

She did note that were the EPSB to take action on any matter, it would be leveled at individuals and not at the district as a whole. The EPSB handles the issuance of certification for all teachers and administrators in Kentucky.

Shelley Johnson with the Kentucky Attorney General’s Office said her office has not yet received a referral from KDE about the matter, while a message left with the U.S. Dept. of Education’s Office of Inspector General was not returned.

Grady Varney, chairman of the Hazard Independent board of education refuted an editorial published in the January 19 edition of the Hazard Herald which includes a comment Varney made during the January 13 regular meeting. Following a discussion about the hiring of a new ACT coach at Hazard High School, Varney quipped, “We will not have any extra money for erasers or anything like that.”

ACT: Not the Only Measure of College Readiness

Council on Postsecondary Education honcho Bob King recently argued in the Herald-Leader that CPE's High School Feedback Report allows the state "to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam."

King suggested that the ACT, by itself, was superior to predictions of college-readiness derived from combinations of data. Discounting graduation rates and average GPA in favor of a single test prompted our resident testing expert to retort.

NOTE: H-L seemed to struggle editing Skip's piece, so here's the unadulterated article the paper titled:

ACT not key measure of college readiness

By Skip Kifer

If I were to assert that a player who cannot make 56% of his free throws is not "ready" for the NBA, a fan would point out that there is much more to basketball than shooting free throws. An astute fan with a historic prospective would point out that Wilt Chamberlain, Shaquille O'Neal and a bevy of other current players would not be "ready" using that arbitrary standard. One facet of basketball does not make or break a player's "readiness."

Bob King, president of Kentucky's Council on Postsecondary Education, in a recent op-ed piece applies the council's arbitrary standard of using a single test score to determine whether a student is "ready" for regular course work in Kentucky's public universities. He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's "readiness" for higher education.

Decades of research indicate: performance in academic courses in high school is the single best predictor of success in higher education; a combination of the high school record and test scores predict better than the high school record alone; and, how good the prediction is and how the components are combined vary depending on the institution. Although there is a general pattern of the primacy of the high school record, there is no one-fits-all model to predict grades in different courses or different institutions.

In addition to the thoroughly suspect notion of labeling a test score readiness, the council's use of a single score for placement purposes violates standards for the proper use of tests. Those standards include the following:

In educational settings, a decision or characterization that will have major impact on a student should not be made on the basis of a single test score. Other relevant information should be taken into account if it will enhance the overall validity of the decision.

In addition:

When test scores are intended to be used as part of the process for making decisions for educational placement, promotion, or implementation of prescribed educational plans, empirical evidence documenting the relationship among particular test scores, the instructional programs, and desire student outcomes should be provided. When adequate empirical evidence is not available, users should be cautioned to weight test results accordingly in light of other relevant information about the student.

Apparently, the council determines readiness by doing statistical analyses of ACT scores and grades in first year courses without regard to institution. A certain ACT score produces a 50/50 chance of getting certain grades, say C, or better. I could find no information about this or other investigations done by the council. And, although it is possible to present information about how good a model is, I could not find any information of that kind either.

To give a sense of the power of statistical models to predict first year grades I report analyses conducted years ago on University of Kentucky student samples. The question was whether results of KIRIS, the first commonwealth assessment related to school reform, could be used for admission and placement in a university.

If a model exactly predicts grades one can say that the model accounts for 100 % of what could be known. If a model cannot at all predict grades, one can say that 0% is accounted for. One way, then, to talk about the power of a statistical model is determine what percent the model predicts.
The table below gives those percents for different courses at UK for three different statistical models: High school record only, High School record + ACT scores and High School record + KIRIS scores.

The first thing to recognize is that the models are not particularly powerful. They rarely account for 25% of what could be known leaving 75% to be explained. That 75% may be differences in students' study habits, class attendance, interests, any of a thousand other variables or simply things not explained statistically.

The pattern of results, however, is clear. Adding an ACT score to a model containing GPA makes the prediction better but not greatly so. The same is true for KIRIS scores, too. Incidentally, that was without including the KIRIS writing sample.

These are not unusual results. They point, obviously, to gathering more information about a student before making a placement decision. Here is what ACT says:

ACT offers a variety of tools to ensure postsecondary students are quickly and accurately placed in courses appropriate to their skill levels. Assessment tools from ACT offer a highly accurate and cost-effective basis for course placement. By combining students' test scores with information about their high school coursework and their needs, interests, and goals, advisors and faculty members can make placement recommendations with a high degree of validity.

To that, I would add, for obvious reasons, it is desirable for an educational agency to use tests in exemplary ways.

The op-ed piece goes on to exhort parents to ask right questions, asks an undefined "we" to fear international test results, says admissions offices should align themselves with the council's readiness standards, and the still undefined "we" to serve teachers more effectively. Such exhortations would be more convincing if, in the first instance, the council could propose placement procedures based on a robust notion of "readiness" that in addition did not violate test score use standards.

Results of ACT Investigation in Perry Co Point to Altered Tests

Students tell ACT Investigators

Proctor Gave Answers During the Test

Supt searches for Explanations

This from the Hazard Herald:

Altered exam responses, multiple handwriting sources and statements from students were all given as evidence by a national testing organization to cancel test scores for several students at two Perry County high schools during an investigation in 2010, according to documents obtained in an open records request by the Hazard Herald.

The findings of ACT’s investigation into allegedly compromised test scores at Perry County Central and Buckhorn High School were noted in letters from ACT to Superintendent John Paul Amis and the high schools’ principals dated November 23.

Though ACT has made no finding of fault by any particular local testing administrator, it does list the organization’s key findings following the investigation which concluded in November, most notably that ACT “found evidence indicating that answer sheets for a significant portion of the students tested at PCCHS were manipulated.” ...

Fifty-nine students from Perry Central took the ACT exam again in October after their initial scores were canceled. According to ACT, the average composite scores decreased by more than one point, and 59 percent of the scores decreased from the initial exam.

ACT investigators found similar aberrations on the 2009 PLAN and ACT exams from Perry Central...

Superintendent John Paul Amis tried to explain the situation to the Herald:

ACT says there were: a high number of erasures from incorrect to correct responses.

Amis says: "...while it wouldn't happen very often, a student taking the test could go back and review answers and find that they were bubbled in out of order or on the wrong page of the answer sheet and have to change them before the time allotted ran out."

ACT says there were: slash marks on answer sheets that forensic handwriting analysts deemed were the result of someone other than the student marking responses on the test.

Amis says: “I don’t think that’s anything abnormal.”

ACT says: an adult proctor provided students with answers during the test,”

Amis says: “I could not find any student that felt like teachers provided answers to them.”

The Kentucky Department of Education forwarded ACT’s findings to three government agencies for further review: the Kentucky Attorney General’s Office, the Education Professional Standards Board and the Office of Inspector General.

Editorial: ACT results give clearer picture of local academics

This from the Kentucky Standard:

The 2009 ACT results are in and while many hoped for gains, the overall state combined score of 18.2 showed a slight decline from the 18.3 posted in 2008. In only its second year of mandated ACT testing for all Kentucky high school juniors, many say it’s too early to look for trends and determine their meaning. However, one thing the ACT results made clear is that Kentucky high school juniors are not ready for college. Based on the 2009 ACT scores, only 46 percent of Kentucky’s juniors are ready for college-level English classes, 21 percent are ready for college-level algebra, 30 percent are ready for science and only 16 percent are ready for biology college courses.

On a local level, Nelson County and Bardstown high schools did well by posting higher composite scores than the state average of 18.2.

Nelson County’s overall composite score was 18.8 with posted increases over 2008 in English, math, science and reading. Bardstown High School posted an overall composite score of 18.3, but saw declines in all four subject areas compared to last year’s test scores.

On the positive side while the state-wide results may not be where we would like them to be at least this is a starting point for improvements. There is much needed work in aligning our high school curriculum with college entrance expectations to better prepare graduating seniors. As Kentucky pushes to make higher education more attainable for our young adults, using ACT mandatory testing as a benchmark of progress is a good first step.

Hat tip to KSBA.

MORE PUBLIC SCHOOL GRADUATES TAKE ACT

The number of public high school graduates taking the ACT increased by more than 14,000 from 2008 to 2009, the Kentucky Department of Education and the Council on Postsecondary Education announced today.

This is the first year in which ACT, Inc. has provided Kentucky with separate scores for public school students for the national data release. The number of Kentucky public school students whose scores are presented in this data was 41,099, a significant increase from 2008’s 26,610.

This increase is due in part to the requirement that all public school 11th graders participate in the ACT. To compile information for the release of graduating class data, ACT, Inc. used students’ scores from the last time they took the test, and many public school 11th graders in Kentucky did not take the ACT again as 12th graders.

(NOTE: The scores in this data release should not be compared to the scores released for Kentucky public school juniors on August 10. The groups represented in the two data releases are not comprised of the same students. Scores from this data release also should not be compared to previous years’ data because of the addition of more students and because of the inclusion of non-public students’ scores in earlier data collections.)

The 2009 composite for Kentucky public school students is 19.1, compared to 2008’s composite of 20.6. Nationally, the 2009 composite score was 21.1, the same as that for 2008. The national composite score includes both public and non-public school test-takers.

ACT indicates that a larger pool of test-takers tends to lead to lower overall scores.

“The good news is that we now have identified hundreds of additional students who, in previous years, would not have been identified as having college aspirations,” said Education Commissioner Terry Holliday. “Our goal is to prepare ALL of our children for career and postsecondary work. While we have work to do, the alignment of Senate Bill 1, Race to the Top and numerous other Kentucky reform efforts are now focused on the right stuff – preparing children for their future, not our future.”

“These test results emphasize the importance of Senate Bill 1,” said Robert King, president of the Council on Postsecondary Education. “SB1 compels a critical partnership between P-12 and higher education for the purpose of increasing the number of young people who attend college, persist and earn a degree.”

Senate Bill 1, passed in the 2009 session of the Kentucky General Assembly, calls for a complete revision of the state’s assessment and accountability system for public schools. The bill’s provisions include a revision of academic standards to be based on national and international benchmarks with the goal of increasing the rigor and focus of subject-area content.

The overall ACT Assessment consists of tests in four areas: English, mathematics, reading and science reasoning. ACT, Inc. recommends that college-bound students take four or more years of English; three or more years of mathematics (including Algebra 1, Algebra 2 and geometry); three or more years of social studies; and three or more years of natural sciences. ACT, Inc. defines specific courses in these areas. Kentucky's graduation requirements define four credits in English; three in mathematics; three in social studies; three in science; one in history and appreciation of visual and performing arts; and one-half each in health and physical education.

ACT, Inc. developed College Readiness Benchmarks in English, mathematics, science and reading, with research indicating that students who reach those have a 50 percent chance of obtaining a B or higher or about a 75 percent chance of obtaining a C or higher in the corresponding credit-bearing college course. The benchmark scores are:

§ 18 or higher on the ACT English Test
§ 22 or higher on the ACT Mathematics Test
§ 21 or higher on the ACT Reading Test
§ 24 or higher on the ACT Science Test

Percentages/Numbers of Kentucky Public School Students
Meeting ACT College Readiness Benchmarks

Some of the drops in percentages of Kentucky public school students meeting the benchmarks can be attributed to the larger population of students who took the ACT and to some students’ status as 11th graders who have not yet completed all of the credits necessary for graduation.

Many Kentucky colleges and universities use ACT scores to inform admissions decisions and to place students in appropriate college courses. ACT scores also are used, along with high school grade point averages, to determine the amount of money high school graduates are eligible to receive through the Kentucky Educational Excellence Scholarship program.

In comparing Kentucky with the nation, the widest gaps in performance among students were in overall English and mathematics scores (2.3 points), and the smallest was in science (1.5 points).

Kentucky’s African-American public school students’ average composite score was 0.6 points lower than the national average for African-American students. At both the national and state levels, the gap between the performance of African-American and white students persisted.

AVERAGE KENTUCKY PUBLIC SCHOOL

ACT COMPOSITE SCORES BY ETHNICITY -- 2005-2009

The composite score gaps between public school males and females of all ethnic groups in Kentucky were minimal in most subjects, with males posting a composite score of 19.0 and females a score of 19.2. On ACT, which offers only multiple-choice questions, males tend to outscore females in mathematics and science, and females tend to outscore males in English and reading.

Kentucky is one of only a few states that has implemented the Educational Planning and Assessment System (EPAS) from ACT, Inc. and, through 2006’s Senate Bill 130, administers the ACT to all public school juniors. The state assesses public school 8th graders using the EXPLORE test and assesses public school 10th graders with the PLAN test through EPAS.

SOURCE: KDE press release

Most Ky. juniors aren't ready for college, ACT results show

Are our 4-year olds ready for 1st grade? College readiness is an important issue for Kentucky, but when I was high school junior, I wasn't ready for college either. I'm just sayin'.

Here some background on the ACT from KSN&C:

Skip Kifer's pieces - To Predict or Not to Predict and A Narrow View of Student Potential

Ben Oldham's piece - Adding Understanding to ACT Scores

And my little rant - What the Bluegrass Institute Doesn't seem to know About the ACT

Susan Weston Spanks the ACT's PLAN test here and CPE President Robert King here.

This from the Herald-Leader:

Kentucky's public high school juniors improved their math scores slightly on the ACT this year, but scores on the test in other subjects remained flat or fell slightly.

The 2009 ACT scores also show that less than half of public high school juniors in Kentucky are ready to do college-level work in English, algebra and other subjects...

Overall, state ACT scores changed little from 2008. The report suggests no dramatic overall trends, because this is only the second year that all juniors in Kentucky public schools have been required to take the ACT. The test assesses students' ability to complete college work and is considered the most widely accepted college entrance exam.

Kentucky is one of only a handful of states requiring the test, which assesses English, reading, mathematics and science skills. Each subject is scored on a scale of 1 to 36. The test is administered statewide on the same day.

Monday's ACT report drew a variety of responses from educational experts around the state.

Richard Day...said that with only two years of scores, it's too soon to declare a trend. Even if a trend were spotted, Day said, it would be tough to assign a meaning.

"What would you attribute that to? The ACT curriculum is not what Kentucky teaches," Day said.

Rather, Day said, the ACT is one of several measures for evaluating student performance, designed to distinguish "between good students and very good students." ...

...Lisa Gross, spokeswoman for the state education department, said the relatively low level of college preparation might reflect that many juniors hadn't taken upper-level courses yet.

Robert King, president of the Kentucky Council on Postsecondary Education, said that the state's educational bureaucracy is trying to figure out how to align high school curricula so that students will arrive at college better prepared. The step is one of several educational changes required under legislation passed by the General Assembly this year.

As things stand now, King said, Kentucky students can do everything that's asked of them in K-12 and not be competitive in college. The goal, he said, is for colleges to communicate what they expect incoming students to know, and for Kentucky's elementary and secondary education system to teach it...

...Yvonne Baldwin, an administrator at Morehead State University, cautioned against putting too much stock in ACT results.

"Relying on the ACT as the sole measure of college readiness is a trap, because it gives too much power to the test," Baldwin said. "I think we need to go back to teaching high school in high school, and college in college. ... The national trend data says that a rigorous high school experience is the best indicator of college success. And I don't think that's happening for a lot of students."

ACT RESULTS FOR PUBLIC SCHOOL JUNIORS RELEASED

Overall results from the 2009 administration of the ACT to Kentucky’s public school juniors show a small improvement in mathematics, but minor drops or flat scores in other subjects.

As mandated by KRS 158.6453, all of Kentucky’s public school juniors participate in the ACT, which assesses English, reading, mathematics and science and is scored on a scale of 1 to 36. The cost of the exam is paid for by state funds. In spring 2009, 43,511 public school juniors took the ACT.

KRS 158.6453 mandates that Kentucky’s public school students participate in the Educational Planning and Assessment System (EPAS) from ACT. The state assesses public school 8th graders using the EXPLORE test, public school 10th graders with the PLAN test and public school 11th graders through the ACT.

Senate Bill 1, passed in the 2009 session of the Kentucky General Assembly, mandates that ACT results be included in school and district accountability results in the 2011-12 school year.

ACT provides information about high school students who take recommended core courses, which include four years of English, three mathematics courses, three social studies courses and three science courses over the four years of high school. Students who indicate that they are taking the core courses or more generally score at higher levels on the ACT.

2009 KENTUCKY JUNIORS’ COURSE-TAKING PATTERNS/ACT COMPOSITE SCORES

ACT developed College Readiness Benchmarks in English, mathematics, science and reading, with research indicating that students who reach those have a 50 percent chance of obtaining a B or higher or about a 75 percent chance of obtaining a C or higher in the corresponding credit-bearing college course. The benchmark scores are:

§ 18 or higher on the ACT English Test
§ 22 or higher on the ACT Mathematics Test
§ 21 or higher on the ACT Reading Test
§ 24 or higher on the ACT Science Test

For the Kentucky public school juniors who took the ACT assessment in 2008 and 2009:

Score information for schools and districts may be found here.

SOURCE: KDE press release

Frankfort to Aim at Illusion

A couple of weeks ago Frankfort Superintendent Rich Crowe said of the lack of accountability for writing under Senate ill 1, “If they’re not going to score it, and if it’s not going to count for anything, I don’t know why we should waste our time.”

Now liberated from state imposed accountability system he is building his own; and he's pointing his teachers toward a target that doesn't quite exist. Crowe told the Frankfort Independent Board of Education in March that he’s inclined to focus on curriculum areas that actually count toward the ratings. Then he chose the ACT.

KSN&C contributor Skip Kifer recently noted,

ACT produces results presumed to help students learn and school personnel make better decisions. Does it? What is the evidence that the information is used? What is the evidence that using the evidence makes a difference? On what?

The problem is that there is no fixed curriculum underlying the ACT that teachers can be assured will be on the test. If students are taught a skill, and perform well on items that test that skill, the ACT throws the test items out in favor of something the teachers haven't taught so well.

This from the State Journal:

New curriculum will center on ACT success

Frankfort Independent teachers will stay in the classroom a week longer than their students this summer, building a new curriculum aimed at success on the ACT college entrance exam.

Teachers will spend June 15 to 19 in subject-related “professional learning communities,” Superintendent Rich Crowe said Thursday at a meeting of the Board of Education.

The goal is to base a kindergarten through 12th grade curriculum on the concepts students must know to reach national benchmark ACT scores – the minimum score required to avoid remediation courses in college...

So despite broad objections from teachers and parents about schools wasting time teaching for the test, Frankfort Independent is getting ready to teach to a different test - one with an elusive target.

One wonders what curriculum the teachers will focus on?

Evidence suggests that the best preparation for doing well on the ACT includes having ones students come from a family that has resources and values education. I'm not sure how the faculty is going to work that out.

Will Frankfort teachers be held accountable if students don't "measure up?" What does it mean to measure up on a benchmark test like the ACT? ...score at the 50th percentile? ...60th? ...40th?

As Kifer notes,

Benchmark tests may or may not be related to what students have been taught. Alignment studies have to be conducted to determine if they are and to what extent.

Since the questions are not released, a teacher does not know what has been done well or poorly. Students have no idea how well they are doing because the questions and the right answers are never discussed. There is no way to tie the results to what a teacher has been teaching because the test, at that time, does not necessarily reflect what has been covered in the curriculum.

Ben Oldham recently wrote that the promoters of the ACT have been over-interpreting its results - making claims that just haven't been shown to be true. But when pressed to prove their points, the promoters go away. Such misuse has even led The National Association for College Admission Counseling Commission to question the test's use by colleges. To the extent that colleges quit using the ACT, its only utility goes away.

Kifer warned folks not to get snookered by ACT officials new claims that without having changed the nature of the test, their scores now tell whether a student "meets expectations" or is "ready" to attend college.

Crowe doesn't seem to be listening.

To Predict or Not to Predict

By Skip Kifer

A colleague and I were doing a session on formative assessment that began with, would you believe, a formative test on creating and using formative assessments. We generated a lively discussion when going over answers to the test, particularly the answer to this question:

Which of the following is NOT a purpose for using formative assessments?

a. To learn which students are doing well and which are not doing so well.
b. To gather data about what has been taught well or not so well.
c. To predict future performance on a norm referenced test.
d. To provide a basis for future instruction.

C is the correct answer. Some teachers insisted that C was not a correct answer because prediction, too, was a purpose of formative assessments. They used as examples what they have learned recently about “benchmark” assessments.

There is no one definition of formative assessment. And there are those, mainly vendors like ACT, who say that benchmark assessments are formative assessments. They are not.

The crux of the difference between benchmark assessments and formative assessments is the difference between predicting and correcting. Benchmark assessments predict; formative assessments correct.

I give formative tests after teaching an instructional unit. The tests:

a. cover what has been done in the unit;
b. are scored but not graded;
c. are discussed immediately upon completion; and
d. are discussed (hopefully) in earnest by the students.

They are now, therefore, instructional tools. They convey to the student what was important in the unit. The students convey what they learned. My aim is to get most students to know most of the material in the unit. In doing so, I want high scores but small differences, less variance, in the scores. I want each student to know more and the class to be more alike.

The discussion is the first part of correcting. By getting a conversation going, I hope that students better understand what I thought I was teaching. The next step, however, is to look at the students’ scores and figure out what additional experience each student needs in order to master the material. I also peek at the scores to see what I did well and not so well.

Contrast the formative assessment scenario with a benchmark one.

Benchmark tests may or may not be related to what students have been taught. Alignment studies have to be conducted to determine if they are and to what extent.

Since the questions are not released, a teacher does not know what has been done well or poorly. Students have no idea how well they are doing because the questions and the right answers are never discussed. There is no way to tie the results to what a teacher has been teaching because the test, at that time, does not necessarily reflect what has been covered in the curriculum.

What benchmark tests do is predict scores on other tests. That is, benchmark results are used to estimate the extent to which student scores on a first test are replicated by results of a second test. That prediction is strongest when the variation on the tests is largest. On the other hand, the more similar are student scores, the worse the prediction. If each student did exactly the same on the first test, the prediction would be zero. So my attempts to use formative assessments to get high scores with small differences between them fly in the face of strong prediction.

Imagine a student entering a classroom and the teacher saying she can predict where the student will be at the end of the year. The student will be about as much above the mean or below as they now are. That strong prediction would be desirable from a benchmark point of view. It may not be so desirable from the student’s view. But that’s predicting!

Imagine that same student entering a classroom and a teacher saying we are going to work together to master the material in this course. She will use formative assessments to help the student. That would imply no relationship between the status of students upon entering the class and their final status in the class. That is correcting. And, teaching!

Just for fun here is the formative test on formative assessment and an answer key.

A formative assessment on formative assessment…..

1. The person who first distinguished between formative and summative approaches was:

a. Paul Black
b. Benjamin Bloom
c. Michael Scriven
d. Richard Stiggins


2. When the chef tastes the soup, it is _____? When the customer tastes the soup, it is _____? Choose the two words that best complete these thoughts.

a. summative, formative
b. formative, summative
c. objective, formative
d. summative, objective

3. Which of the following is NOT a purpose for using formative assessments?

a. To learn which students are doing well and which are not doing so well.
b. To gather data about what has been taught well or not so well.
c. To predict future performance on a norm-referenced test.
d. To provide a basis for future instruction.

4. Formative assessment activities in the classroom used properly:

a. Provide a basis to rank order students.
b. Predict performance on other tests.
c. Help the teacher know what she has done well or poorly.
d. Add to the precision and usefulness of grades.

5. The Taxonomy of Educational Objectives:

a. Orders outcomes according to how difficult it is to teach them.
b. Is based on learning hierarchies
c. Shows what is desirable
d. Is ordered by cognitive complexity

6. Suppose students in a class were asked to memorize the names of those who came over on the Mayflower. A test question then asked them to write down 10 of those names. At what level of the Taxonomy is the response likely to be?

a. Knowledge
b. Comprehension
c. Application
d. Analysis

7. Suppose the list contains occupations of the Mayflower passengers. Students are asked to write an essay that draws inferences about why certain occupations were present on the ship. At what level of the Taxonomy are the responses likely to fall?

a. Knowledge
b. Comprehension
c. Application
d. Analysis

8. Suppose the list of names of persons on the Mayflower also included their occupations. Students are asked to write an essay that draws inferences about why certain occupations are on the ship. That would represent what Depth of Knowledge?

a. DOK 1
b. DOK 2
c. DOK 3
d. DOK 4


Key to Formative Test

1. C. Scriven, M. (1967). The methodology of evaluation. Tyler, Gagne & Scriven (Eds). Perspectives of curriculum evaluation. (AERA Monograph Series on Curriculum Evaluation, No. 1). Chicago: Rand McNally
2. B. Bob Stake coined this to distinguish between the two.
3. C. Prediction comes from benchmark assessments. A teacher would like all of her students to do well on the formative tests. If so, there would be no correlation (prediction) between the formative test results and some norm referenced test results.
4. C. First of all, did the teacher do well, on what? Secondly, which students did well and which did poorly, on what?
5. D. Bloom called the Taxonomy the most used, least read book in education. The taxonomy orders outcomes according to cognitive complexity.
6. A. Recall would fall under Knowledge, the least complex cognitively of the levels.
7. C or D. A taxonomy purist would probably say D. I have difficulty getting beyond Applications.
8. C or D. I don’t do Depth of Knowledge. I don’t think it adds anything to the taxonomy. There are lots of classification schemes. I am happy being familiar with one.

Don't use SATs to rank college quality

This Op Ed from Director of Public Policy and Research for the National Association for College Admission Counseling David Hawkins at CNN.com:

ARLINGTON, Virginia (CNN) -- A recent controversy at Baylor University has brought new attention to the widespread misuse of standardized college admission tests to rank the quality of America's colleges and universities.

The SAT and ACT have achieved nearly iconic status in America and throughout the world. As the late author and scholar Stephen Jay Gould noted, the tests are viewed by many as "abstruse and mysterious," with powerful effects on public policy, social mobility, and even individual identity.

Misuse of test scores as college ranking criteria creates undue pressure on admission offices to pursue increasingly high test scores and fuels the tests' disproportionate influence...

...It's critical to understand that high SAT and ACT scores alone do not equate to merit. Last month, our organization's Commission on the Use of Standardized Tests in Undergraduate Admission issued a report calling for change in the way everyone
involved in college admissions uses the tests....

...Test misuses feed the public perception that the scores are the most influential factor in college admission....

Thinking about the CATS Writing Portfolio and the ACT

At last week's CATS Task Force meeting there was a suggestion that Kentucky - being the only state to use writing portfolios in its state assessment - is surely out-of-step with other, more enlightened states.

After all, if writing portfolios were such a good idea, everybody would want them. Right?

Supposedly to fix this problem, those who want to discontinue the problematic CATS suggest substituting the ACT, with its over-advertized benchmarks.

But lo and behold, Kentucky is one of just five states that requires the ACT for all high school juniors.

I'm just sayin' ...

Kifer on the Problem with an ACT-only Assessment

KSN&C has maintained that what's really wrong with Senate Bill 1 and its suggestion of using the ACT as THE Kentucky test, is that using the ACT doesn't fix the problem. As one part of a comprehensive testing program the ACT may have a place. But alone, it fails.

Sunday, Skip Kifer illustrated the point in the Herald-Leader.

Narrow view of student potential
By Skip Kifer

We old guys remember what the ACT did in its heyday. Along with a student's high school record, ACT scores helped admissions offices decide who had the best chance of succeeding in their institutions. Colleges used ACT scores, along with an array of other information, to place students in courses once they arrived on campus.

Last month, a committee representing the nation's admissions officers released a report about overemphasizing test scores and argued for prudent use of the ACT and the Scholastic Achievement Test in selecting students. The committee suggested that some colleges and universities can enroll students without requiring the tests. This is a recurring theme, not a new one.

What is new is that ACT officials, without having changed the nature of the test, say their scores now tell whether a student "meets expectations" or is "ready" to attend college.

Kentucky's high school juniors are required to take the ACT. Their results, released a couple of weeks ago, were interpreted to mean that too many students failed to "meet expectations" or too few were "ready."

What, we old guys wonder, could these assertions mean?

An ACT score is just that: a score on a test. It tells something about me as a student but means little without a context.

My 18 on the ACT means something very different if I am in the top 10 percent of my high school graduating class rather than just in the top half. My 18 means something different if I have taken solid courses rather than choosing an easy way to a diploma.

My 18 means something different if I were to take the test again. My 18 on the ACT means something different depending on the college or university I attend. Among Kentucky's public universities, for example, my 18 would be a relatively low score in one but a relatively high score in another.

My 18 is like Shaquille O'Neal's free-throw percentage. Being able to shoot barely 50 percent from the free-throw line says something about O'Neal. But, does it mean not "meeting expectations" or not being "ready" for the NBA? Hardly. NBA scouts are clever people who would not let one piece of information become a judgment.

ACT says that a person with an English score of 18 has a 50/50 chance of getting a B or higher in a college English course. We old guys say that is throwing the statistical dice.

My left foot is in boiling water and my right foot is in ice water but "on the average" I feel fine.

Of all students with a score of 18, one-half will get a B or better and one-half a C or worse. In which half is my 18? Is it in the ice or boiling water? ACT cannot answer that question. But if I were in the top 10 percent of my class, it is more likely that I will get the B or better.

We old guys also wonder why so much was made of ACT results and so little reported about Kentucky's Advanced Placement results. As opposed to the ACT, AP scores are a direct measure of whether a student can do college work.

Students take AP courses in a variety of subjects. An AP course is so well defined that what Kentucky students experience is comparable to what is experienced by students throughout the country. Each AP course is taught by a capable teacher and is described in detail with course goals, materials and examinations available for scrutiny.

The examination measures what is learned in the course. High-school students who score 3, 4 or 5 on an AP examination can get placement or credit or both in the college or university they attend.

Recent results show increased numbers of Kentucky's students taking AP courses and scoring higher on them. That is, more Kentucky students are doing acceptable college work in high school. It would make sense, therefore, for an educational community wanting more students to attend college to work to expand AP opportunities.

While it is important to provide opportunities to prepare each student to take AP courses, it is unlikely that all students will take them. Even so, AP courses point in a positive direction for schools. They are a model for structuring courses and tying together testing and instruction.

An algebra course, for example, in Fayette County should be the algebra course in Christian County. More important, two algebra courses in one Fayette County school should be the same algebra course.

AP results show, not surprisingly, that students tend to learn what they are taught. It is important, therefore, to define and teach what it is important to learn. Then one can assess and give credit.

Someone might ask the old guys what this has to do with O'Neal. The answer is clear. If you want to know what kind of basketball player he is, you let him play basketball. You don't look only at his free-throw shooting percentage and decide that he does not meet expectations.

Study of Standardized Admissions Tests Is Big Draw at College Conference

This from the New York Times:

[At] the National Association for College Admission Counseling, the main event was William R. Fitzsimmons’s first public presentation of the findings of the Study of the Use of Standardized Tests in Undergraduate Admission.

Mr. Fitzsimmons, the dean of admissions at Harvard, led a commission of college admissions officials who drafted the study, which challenges colleges and universities to examine their use of the SAT and ACT and to consider whether the benefits outweigh the disadvantages or whether they can make the tests optional for admissions.

The line formed early for Mr. Fitzsimmons’s panel, and with more than 1,000 people jockeying for a limited number of seats — a scene that brought to mind the college admissions process — the event was moved to the ballroom.

“It’s electrifying to both sides of the desk,” said Louis L. Hirsh, admissions director at the University of Delaware, “to counselors who are worried about the stresses that the SAT places on the kids, and from the college end, the people whom all of us respect are looking at a test that all of us use and asking all of us to be more thoughtful about how they use it and what role it plays in our admissions.”

Mr. Fitzsimmons, who took center stage along with the other members of the commission, tried to ease the fears of the ardent supporters of the standardized admissions tests, taking pains to say that the SAT had many advantages.

But he also affirmed what many of those present had been saying for years: that the SAT and other standardized admissions tests are “incredibly imprecise” when it comes to measuring academic ability and how well students will perform in college. He said colleges and universities needed to do much more research into how well the tests predict success at their individual institutions...

...There has been longstanding debate and concern about the impact of standardized testing on socioeconomically disadvantaged students, and the ballroom erupted in applause when Mr. Fitzsimmons called for an end to the use of “cut scores” to determine who qualifies for National Merit and other scholarships. The practice means that one student is rewarded while excluding another whose SAT score may be only a single point lower, Mr. Fitzsimmons said.

What that single point differential fails to take into account, he said, is the context: The two students may have “lived entirely different lives, had entirely different educational opportunities and entirely different access to test prep.”

The audience also applauded Mr. Fitzsimmons’s call for U.S. News & World Report to stop using SAT scores as part of its college rankings.

“At Harvard we get terrific students, and we turn out terrific students later on,” Mr. Fitzsimmons said. “Is that due to Harvard or is that due to the students to begin with? Who knows? There are fabulous institutions with relatively low test-score averages that are absolutely first rate, that take students from point A to point Z.”

He continued, “Educational quality has nothing to do, or very little to do, with actual average SAT scores.” ...

...Jeffrey Brenzel, the dean of admissions at Yale, said the report raised key questions for every college and university: “Are you using the tests in a responsible manner and in the way they were intended? Is your use of the test relevant to your particular institution’s mission? Are there alternatives?” ...

What the Bluegrass Institute Doesn't Seem to Know about the ACT

There is no way to sugar coat this.

Somebody doesn't know what he's talking about

- and it's not Oldham.

Well, Ben Oldham and I got called out by Richard Innes of the Bluegrass Institute the other day for "ignoring" the ACT benchmarks scores - apparently the holy grail of assessment in his mind.

Of course, this is all part of a larger conversation about Senate Bill 1 and the on-going Task Force on Assessment at KDE.

I wasn't planning on getting into all of this this fall. It's tedious, inside baseball kind of stuff. But the fundamentals are still the same. First, it's just a test. Second, every test has been designed to do a specific job. If test designers wanted a test to do something else, they would begin with that fact in sight. Third, have I mentioned it's just a test?

OK, let's talk about the ACT.

Here's the problem with the American College Test: Nothing, really.

It is a well-designed test intended to help admissions officers at competitive colleges determine which students are most likely to be successful at the university level. Ben Oldham recently went further saying, the "American College Test (ACT) is a highly regarded test developed by a cadre of some of the best measurement professionals in the world and is used by a number of colleges..."

But the ACT is only ONE factor that colleges use to make such determinations.

Why?

I mean, if the ACT can predict success in life, as Innes un-credibly argues (below), why don't colleges simply rely on it and quit wasting time compiling grade point averages and other data they say they need to made the best choices for their school?

The answer lies in the fact that test data are only reliable up to a point. It's just a test score and it shouldn't be turned into anything more.

As Richard C. Atkinson and Saul Geiser recently pointed out,

the problem with general-reasoning tests like the SAT [and ACT] is their premise: that something as complex as intellectual promise can be captured in a single test and reflected in a single score. It is tempting for admissions officers--and parents, legislators, policymakers and the media--to read more into SAT [and ACT] scores than the numbers can bear. Although measurement experts know that tests are only intended as approximations, the fact that scores frequently come with fancy charts and tables can create an exaggerated sense of precision.

And such exaggerations persist.

Newspapers and bloggers rank scores that ought not be ranked - because people like rankings. Some "think tanks" act as though test scores equal "truth" and look for any opportunity to twist data into a pre-existing narrative that Kentucky schools are going to hell in a handcart - this, despite trend data to the contrary, about which they are in full denial.

Georgetown College Distinguished Service Professor Ben Oldham correctly warned that, "since the ACT is administered to all Kentucky juniors, there is a tendency to over-interpret the results as a measure of the success of Kentucky schools. His excellent article clarifies what the test is, and what it isn't.

The problem of over-interpretation has been somewhat exacerbated by the inclusion of benchmark scores in the ACT. But benchmarking does not change the construction of the test nor the norming procedures. It does not turn the ACT into a criterion-referenced exam as Innes tries to suggest - unless all one means by "criterion" is that the ACT derived a cut score. Under that definition a People Magazine Celebrity Quiz could be considered criterion. Socre 18 and you're a Hollywood Insider!

The ACT's "criteria" simply does not measure how well Kentucky students are accessing the curriculum. It is much more sensitive to socio-economic factors attributable to most of the college-going population.

Using a "convenience sample" of schools (those willing to participate) the ACT looks at student success in particular college courses; and then looks at the ACT scores obtained by "successful" students. But regardless of what data such a design produces, "there is no guarantee that it is representative of all colleges in the U.S." Further the ACT "weighted the sample so that it would be representative of a wider variety of schools in terms of their selectivity."

That is to say, they tried to statistically adjust the data produced by the sample to account for more highly selective schools as well as the less selective. This process of weighting data to produce a score that the original sample did not produce should be viewed suspiciously. It would be like...oh, let's say like....using a concordance table to give students a score on a test they didn't take.

If KDE had done anything like this, Innes' buddies at BGI would be crying "fraud."

If we are going to test, and if our tests are going to be used to determine placement in programs within schools, and eventually in college, then we need to understand what the ACT means when it says "college-ready." And we don't. The most important flaw of the ACT benchmarks is conceptual: What is "readiness" for higher education?

As one delves deeped into the statistics other problems arise. Skip Kifer who serves on the Design and Analysis Committee for NAEP told KSN&C,

The benchmark stuff is statistically indefensible. Hierarchical Linear Modeling
(HLM) was invented because people kept confusing at what level to model things
and how questions were different at different levels. The fundamental
statistical flaw in the benchmark ... is that it ignores institutions. Students
are part of institutions and should be modeled that way.

But the ACT models at the "student" level when it should be modeling at the "students nested within institutions" level.

It is possible that the ACT took a kind of average of those "correct" models but that can not be determined that from their Technical Report.

Perhaps Innes could help us understand: How is it that the ACT's benchmarks could have been empirically defined and yet managed to get the same relationship for the University of Kentucky and Lindsey Wilson College?

Unfortunatley, the ACT folks did not respond an inquiry from KSN&C.

But none of this will likely stop the exaggeration of the ACT's abilities.

In response to a KSN&C posting of Ben Oldham's article, Innes made the following claim:

Oldham pushes out-of-date thinking that the ACT is only a norm-referenced test. The ACT did start out more or less that way, years ago, but the addition of the benchmark scores, which are empirically developed from actual college student performance to indicate a good probability of college success, provides a criterion-referenced element today, as well.

"Criterion-referenced element?!" A cut score? The ACT is a timed test too - but that doesn't make it a stopwatch.

So, Oldham is old fashioned and out-of-date? Au contraire. It is Innes who is over-reaching.

Innes argues,

the ACT says that many employers for ... better paying jobs now want exactly the same skills that are needed to succeed in college. So, the Benchmark scores are more like measures of what is needed for a decent adult life. Thus, it isn’t out of line to say that the Benchmarks can fairly be considered a real measure of proficiency. And, that opens the door to compare the percentages of students reaching EXPLORE and PLAN benchmarks to the percentages that CATS says are Proficient or more.

Bull.

One could derive as much "proficiency" evaluating Daddy's IRS form 1040 and then comparing percentages of students reaching EXPLORE and PLAN benchmarks to the likelihood of owning a BMW or affording cosmetic surgery.

I'm afraid what we have here is something other than a nationally recognized assessment expert who is out-of-date.

We have a pundit who thinks the ACT benchmarks constitute a criterion-referenced assessment of the performance of Kentucky students and their prospects for a decent adult life!? This, absent any connection between the ACT and Kentucky's curriculum beyond pure happenstance. There is no relationship between a student's ACT score and any specified subject matter - which is typically the definition of a criterion-referenced test.

There is no way to sugar coat this. Somebody doesn't know what he's talking about - and it's not Oldham.

The best spin I can put on this is that Innes got snookered by ACT's marketing department, which seems to do a fine job, but has been known to overstate the abilities of ACT's EPAS system.

But none of this makes the ACT a bad test. It just means that assessment experts have to take care to understand the nature of the exams and not to rely on them to do too much.

And it is commendable that Kentucky is working toward building an actual relationship between Kentucky's curriculum and that of the ACT through the development of content tests. That work will get Innes closer to to where he wants to be. He should wait for the actual work to be done before making claims.

Just as Atkison, Geiser, Oldham, Kifer, Sexton and virtually everybody else says, the results should not be over-interpreted to suggest relationships that just aren't there. And trying to argue causal chains that are completely unproven is certainly not best practice.

But more to the point, Kentucky recently committed to use the ACT's EPAS system including EXPLORE and PLAN as yet another measure - a norm-reference measure - of student performance. As long as Kentucky is cognizant of the test's limitations we ought to strengthen the connections between Kentucky high schools and universities and gauge student readiness for college. It was because of the large numbers of college freshmen in need of developmental courses that CPE pushed for the ACT/EPAS system to begin with.

Kifer wonders why Kentucky's Advanced Placement (AP) Tests receive so little attention. After all, unlike the ACT, the AP tests are a direct measure of a high school student's ability to do college work; AP courses are particularly well-defined; the tests exist across the curriculum; good AP teachers abound; course goals and exams are open to scrutiny.

When a high schooler passes an AP test he or she not only knows what it means, but the school of their choice gives them college credit for their effort.

Aware of CPE's commitment to the ACT as one measure of student readiness, KSN&C contacted newly named Senior Vice President of the Lumina Foundation Jim Applegate, who until recently served as CPE's VP for Academic Affairs.

Here's what Jim had to say:

Richard,

The article recently referenced in your publication from the admissions officer group addresses the use of ACT for college admissions. The
organizations sponsoring assessments such as ACT, SAT, and others have made clear that no single standardized test should be used to make such decisions. Postsecondary institutions, to implement best practice, should use a
multi-dimension assessment to make admissions decisions. A test score may play a
role in these decisions, but not the only role.

Kentucky uses the ACT/EPAS system (the Explore, Plan, and ACT tied to ACT ‘s College Readiness Standards) to help determine college readiness, place students in the right high school courses to prepare them for college, and place them in the right courses once they go to college. Kentucky’s revised college readiness standards are
about placement, not admission. For the foreseeable future, the postsecondary
system will, as it has always done, accept large numbers of students with ACT
scores below readiness standards, but will provide developmental educational
services to these students to get them ready for college-level work. The large
number of underprepared students coming into Kentucky’s postsecondary system led the Council a couple of years ago to initiate an effort to improve developmental
education order to make sure these students receive the help they need to
succeed in college.

A growing number of states are adopting the ACT or the entire EPAS system to more effectively address the challenge of getting more high school graduates ready for college or the skilled workplace (e.g., Colorado, Illinois, and Michigan). These states also want to better understand the performance of their students in a national and international context. Globalization no longer allows any state’s educational
system to remain isolated from these contexts.

The use of ACT/EPAS is, of course, only one necessary strategy to improve the college/workplace readiness of Kentucky’s traditional and adult learners. Kentucky is working to implement statewide placement tests in mathematics, reading, and English that will be administered to high school students who fall below statewide college readiness benchmarks tied to ACT scores (few states have gotten this far in
clarifying standards to this level). These placement tests will provide more
finely grained information about what students need to know to be ready for
college-level work. We are also working to more strongly integrate college
readiness goals into our teacher preparation and professional development
programs to ensure teachers know how to use the assessments beginning in middle
school to bring students to readiness standards.

The postsecondary system is hopeful the implementation of ACT/EPAS will promote partnerships between postsecondary and high/middle schools to improve student achievement. Some of that has already begun since the first administration of the EPAS college readiness system. For the first time in my time in Kentucky (I grew up
here and returned to work here in 1977) we now know where every 8th grader is on
the road to college readiness thanks to the administration of the Explore. If in
five years the number of students needing developmental education is not
significantly less than it is today then shame on all of us.

Jim Applegate

All of this reminds me of the old Crest Toothpaste disclaimer I read daily while brushing my teeth over the decades.

Crest has been shown to be an effective decay preventive dentifrice that can be of significant value when used as directed in a conscientiously applied program of oral hygiene and regular professional care.

Let's see if I can paraphrase:

The ACT/EPAS system has been shown to be an effective norm-reference assessment that can be of significant value when used as directed in a conscientiously applied assessment program based on clear curriculum goals, direct assessments of specific curriculum attainment and effective instruction from a caring professional.