ACT Can't Deliver Kentucky's End of Course Exams

Updated:

System is not designed to support enough concurrent users

I got a message from a Kentucky high school principal today. This leader of a top school tells me that District Assessment Coordinators received a note this week from KDE indicating that ACT was incapable of handling the number of students scheduled to take online End of Course exams. KDE had planned a "stress test" for November to assure that ACT's system could handle 20,000 students hitting the servers at one time under testing conditions.

The principal observed, "I don’t think the number of student taking the end of course exams has changed much since ACT pitched their program to the state to win the contract."

 

Correctomundo.

This past summer Commissioner Holliday talked tough about ACT's shortcomings in the last go 'round. 

“Trust me. The vendor is hearing about it from me and our team here. If we can’t get this addressed, we will consider a new RFP for a new vendor,” he said. “We can’t get it done by next year. It’s already summer and it sometimes takes six months to get through an RFP process. But if we don’t get assurances by this fall that this has been addressed, we’ll go all paper and pencil next year, and then go to an RFP process to possibly select a new vendor for the 2014-15 school year.”

"It would seem that ACT is pretty much thumbing their nose at Commissioner Holiday," the principal says.

In response, KDE is beginning the "Request For Information" process which will inform the "Request for Proposal" process. As the Commissioner indicated, in state government it's a long road to a new vendor. The earliest a new vendor could be in place is the 2014-15 academic year. But it's more likely to be 2015-16.

To make matters worse, ACT's tests are based on "Quality Core," which is not the same as Common Core, and some complain that it does not match Kentucky's curriculum all that well. Still, a successful test would have produced useful student data.

Here's the note:

END-OF-COURSE STRESS TEST CANCELLED

KDE asked ACT, Inc. to provide a test of the end-of-course online system. The purpose of this assessment was to provide Kentucky educators assurance the online system would work with a large number of concurrent users. 

Last week, KDE was informed by ACT that system upgrades for large scale users are impossible by the spring of 2014.  ACT confirmed the current system is ultimately not designed to support 20,000 concurrent users. Therefore, the Stress Test scheduled for November 20 would not be a useful activity.  

Plans for the November 20 Stress Test are cancelled. Reasons for the cancellation include, the students would not be able to complete the assessments and summary data could not be provided.  Since the Stress Test would not benefit students, schools, districts or the state, the Stress Test is cancelled.

End of Course exams were provided for under Senate Bill 1 (Unbridled Learning).

This from KSBA in June:

Holliday: If ACT Inc. can’t prove online testing issues fixed,

Kentucky students will take paper & pencil exams

Education Commissioner Terry Holliday continued his tough talk Wednesday about this year’s problems with online, end-of-course testing, telling the Kentucky Board of Education he’s ready to return to paper and pencil tests next year.

“Last school year, we used ACT QualityCore for end-of-course exams online and everything went smoothly,” Holliday said at the outset of today’s KBE meeting. “This year, ACT sold QualityCore to Alabama and Ohio schools. Our vendor had not adequately prepared for the number of students taking the test at the same time.

“We had about 2,000 students who had their online tests interrupted; some by a few minutes, some by two days. All students were able to complete the either online or using paper and pencils. We will ensure that no student, no school and no student is adversely impacted by this interruption with the online testing. We feel we should apologize to these students and teachers for the disruption,” he said.

Holliday, who said last Friday during a KDE webinar that he was prepared to seek a new testing vendor if ACT couldn’t provide assurances it had addressed the capacity issue...

At first blush, paper and pencil End of Course exams would seem to be a dubious solution since they would have to be scored and turned around in time to use as 20% of the students' course grades by the end of the year. However, approximately 120,000 of the states’ 175,000 affected students were tested with paper and pencil last May. Turnaround time is no later than 10 days from the time ACT receives the scores. But it's hard to maintain confidence in ACT at this moment. According to KDE spokeswoman Nancy Rodriguez, “ACT is working on a method to return paper results quicker. 

In addition, there is an issue with math and ACT. As mentioned above, ACT tests Quality Core as opposed to Common Core. As Rodriguez explained, 

There was one standard on the Quality Core Algebra II test in 2012/13 that measured one of the Common Core PLUS standards.  (PLUS standards are common core, but seen as additional standards that would be good to teach, but not necessary). Since Kentucky teachers didn’t have adequate notification and time to adjust their curriculum/instruction to add the PLUS standard, this item was not counted.

Quality Core math only measures a subset of the Common Core State Standards for math and since Algebra II is only one course, it cannot cover all the common core standards required at the high school level. 

It is interesting to note that state test scores dipped so much that the Commissioner mentioned it as a concern when the data were released this fall. Teachers use the Quality Core guidelines and curriculum, which is a subset of the common core math, to design their instruction and curriculum. Was the dip an accurate reflection of poor student performance, or did a curriculum mismatch contribute? 

Turns out the End of Course test is only a reflection of how well the students were taught on the particular subset of common core math concepts that are included by Quality Core. With the demise of EXPLORE and PLAN, soon to be replaced by ASPIRE, is ACT thinking that Kentucky is "small potatoes" compared to larger more financially lucrative states they contract with, or wish to contract with?

One wonders if KDE is planning to recover costs from the company due to its failure to deliver on a bid that the company apparently knew at the time of bidding that it could not satisfy.

ACT has been invited to comment for this story, and if they do, I will update.