Wednesday, April 09, 2014

Ky. orders review after questions raised about use of calculator software on statewide tests

ACT forbids calculators with built-in algebra systems, 
but not those that have had software like Zoom Math added to them!?

This from the Herald-Leader:

The Kentucky Department of Education has ordered a review of how students use advanced calculators on a statewide college-readiness test, citing concerns raised by experts that the devices could be artificially inflating scores.

At issue is the math portion of ACT's COMPASS test, which is used as a placement test for high school seniors who have not met college-readiness benchmarks on the ACT test they take as juniors.
According to ACT rules, which are followed by the state, students are allowed to use certain calculators on the test, including ones that have been loaded with an algebra software program called Zoom Math. ACT forbids calculators with built-in algebra systems, but not those that have had such software added to them.

Stephen Newman, a math professor at Northern Kentucky University, said he is alarmed by the expanding use of Zoom Math, which allows students to plug in algebraic equations and get the correct answer without understanding how to do the math.

In late March, he and some colleagues conducted an experiment to measure the potential impact of Zoom Math on COMPASS test scores. They took the test 10 times, using calculators with Zoom Math to answer all equation problems. On any word problems, they simply chose the multiple choice "A" every time. In all 10 cases, they scored well above the cutoff.

"Based on our experiment, about 55 percent of COMPASS problems can be done without thought using Zoom Math," he said. "This has national implications and it could become a disaster if it continues."

COMPASS assessments are taken by more than 2.2 million students across the U.S. annually in high schools and colleges, which use them for class placement. About 18,000 Kentucky high school students took the COMPASS test in the 2012-13 school year, according to the state Education Department.

ACT is standing firm by its corporate decision to allow use of the add-on software.

"Through ongoing research, ACT ensures that our test administration policies remain fair and consistently enforceable," according to a statement from the Iowa-based company. "ACT math specialists continually review new calculator models, as well as algebraic systems and software, to determine their potential impact on test questions. Our current policies, which prohibit calculators with built-in algebra systems, remain in effect. We are confident that math test scores achieved under our current policies are valid and representative of student achievement."

Kentucky education officials aren't as confident. Ken Draut, director of assessment for the Education Department, has ordered a review to determine if Kentucky needs more stringent rules than those set by ACT.

"We'll be getting together with ACT to discuss our findings no matter what they are," Draut said. "At this point, I'm not worried because I don't think it's widespread, but I want to nip this in the bud for the future. I don't think it has had a big effect yet, but I am worried about the future if this is a program that provides a score we can't consider valid."

According to an Internet search, at least nine Kentucky school districts have purchased Zoom Math. Fayette County Schools do not use the program, but it is used by students on tests in Jefferson County, although the school district would not say by how many schools.

Newman is part of a group of math educators who developed a similar statewide assessment for the Kentucky Department of Education called KYOTE; it is forbidden to use Zoom Math on that test. (Newman said he has no financial interest in the KYOTE test.)

Zoom Math started as a tool to help special education students, said Tammy Herrada, the software company's CEO. Although its software has been around since 2006, Herrada said she heard of its use on ACT tests only in the past year.

The company has never advocated the software as a testing aid, she said.

"It allows students to go over the things they've learned in the past," she said. "We're not selling it to pass any ACT tests. We're selling it to teach and benefit."

Some school officials said its use appears to be expanding around Kentucky.

Cindy Beals, the district assessment coordinator for Warren County Schools, said the district's high school students have access to Zoom Math, but she does not know to what extent they use it on the COMPASS test.

"As long as they are allowed, then we allow our kids to use it," she said.

Tamela Porter, a guidance counselor at Bath County High School, said students use Zoom Math on the ACT and COMPASS tests.

"We were always told that it was fine since KDE had approved it," Porter said. "We use it in daily instruction ... We felt like if we are using it in instruction every day, we don't want to change that once they go take the ACT."

Newman said he doesn't think Zoom Math would have as much benefit on the regular ACT test because there are more analytical word problems.

Still, he would like to see other experts try his experiment and is angry that ACT hasn't moved to ban the software.

"They're wrong," he said. "We cannot continue this fraud. People are suspicious of gains in college readiness. It's our college readiness program that is being threatened."

"Career and college readiness" benchmarks are part of a 2009 overhaul of the state testing system in which ACT products were adopted at nearly every grade level. In 2013, the percentage of students deemed ready for college or a career jumped to 54.1 percent, up from 34 percent in 2010.
However, it's not clear what role, if any, Zoom Math might have had in the 20-point jump, since that score includes results from COMPASS, KYOTE, ACT and other career tests, such as ACT WorkKeys.

At least one other state has already banned use of Zoom Math on statewide tests. Mississippi took that action in 2011 after its education department concluded that Zoom Math "provided students with an unfair advantage and compromised our ability to make specific claims about student learning based on our assessment," said spokeswoman Patrice Guilfoyle.

Bob Schaeffer, a spokesman for the National Center for Fair and Open Testing, said new technology is constantly providing opportunities to give some kids a leg up.

"As high stakes testing moves into the computer area, teachers need to find ways to ensure that some students don't have tools that give them an advantage on the exam," he said.

Donna Caldwell, the district assessment coordinator for Madison County schools, said she heard of Zoom Math only a few weeks ago and doesn't know if individual schools might be using the software on tests.

But she said ACT and the Kentucky Department of Education need to address the issue as the availability of similar technologies expands.

"There's got to be some effect," she said. "Does it make the scores invalid? I don't know that you can say that. Do I think there's a huge impact? No, but a policy has to be developed, and it has to start with ACT."


Anonymous said...

I thought we were suppose to be facilitators, that expecting students to actually possess extensive knowledge was old school. Just give them tools and direction about how to use them within the existing system.

Only reason folks use Zoom Math is to pump up their lower performers scores. What next Zoom Read? Zoom world civ? Zoom SAT? Zoom MCAT/LSAT?

Skip Kifer said...

This is an experience, not an experiment. And it is an experience without valid conclusions.

Two mathematics professor using a calculator and a software program (Zoom Math) answered only the items on the ACT Compass test where the program is applicable. They marked the first answer on the remaining items.

They found their resulting scores exceeded an arbitrary cut-point (college readiness).

They concluded "about 55 percent of the COMPASS problems can be done "without thought" using Zoom Math."

In another setting they said "Our experiment demonstrates that essentially 'any student' who has a calculator equipped with Zoom Math and who has been trained to use it can become college ready in mathematics in Kentucky and will be guaranteed placement into a college-level math class at any public college or university in our state." My italics.

How does one get from two college professors using a calculator and a piece of software to conclusions about not needing any thought to answer questions and that is true for any student with a certain calculator in Kentucky?

That is nonsense.

They could have gotten a good estimate of their results by scoring the test according to answering all the "calculator" items correctly plus adding 1/5 (assuming the answers are assigned at random) of the other items. It would have saved time and made more sense. And one does not need professors using calculators.

There is research about the effects of the use of calculators and software on performance. It is true that under certain conditions, students who use software have an advantage. It may or may not be true about Zoom. But this experience does not answer the question.

Perhaps, it would have been appropriate for these mathematicians to review some of the research, not only for the findings but also how to conduct experiments that yield valid results.