KDE Office of Assessment and Accountability responds to ORQ flap

On February 15th Kentucky School News and Commentary ran a story outing some incorrect training a few educators say they received at the hands of Measured Progress, KDE's testing contractor. Worse, the trainees believed they were being told that Measured Progress used some kind of double-secret rubric that redefiend words and penalized students who followed directions.

KDE responded saying it was all a mistake, that they would follow up and get back to KSN&C with a more formal response. KDE did that today.

This from KDE's Office of Assessment and Accountability, Division of Assessment Support:

POWERPOINT WITH MISINFORMATION CIRCULATING

A PowerPoint has been circulating that concerns formative assessment and answering Kentucky Core Content Test (KCCT) open response questions. This PowerPoint did not come from Measured Progress or KDE. It includes incorrect
information.

Specifically, two slides in the PowerPoint state:

· The question does not always mean what it says. There is hidden meaning in some of the questions. Students must go a step beyond what the prompt requires (Like in the old days of assessment). We must coach students up on this. As a matter of routine, students must do more than what is asked of them by the question prompt.

· List does not mean list. The hidden meaning is list and explain or list and describe.

There are no hidden scoring features in the KCCT items or scoring guides. The question means exactly what it says -- students do not have to go beyond what the prompt requires. List means exactly that -- “list.” If there is a reason to explain or describe, the question will ask students to explain or describe. Answers should always be thorough, complete and full enough to relay knowledge and abilities about the question.

Scoring has not changed under the new contractor, Measured Progress. West Ed (the item/scoring guide contractor since the 1990s) still develops all the items and follows the same guidelines as used in the previous contract. Measured Progress applies the scoring guides created by West Ed to the scoring session.

Please see the recently released annotated open response questions for more details. They provide insight into how items are scored.

For more facts about KCCT scoring....Who ya gonna call?

MYTH BUSTERS

KCCT MYTH:Restating the question is mandatory.

KCCT MYTH BUSTER says: No, it is not required, but is acceptable to do.

MYTH: Responses restating the question without further information will earn at least one point.

MYTH BUSTER: No; new or additional information must be included in order to receive any credit.

MYTH: A graphic organizer should be done on the response page.

MYTH BUSTER: Depending on the type of question being asked, a graphic organizer may not be the best way to record the answer. Best
practice would be to use the organizer on scrap paper to plan the
response.

MYTH: Answers must be in paragraph form.

MYTH BUSTER: Scorers are trained to focus on content and not address the format of the response. A response in any format -- bulleted, labeled diagram or graphic organizer -- will be scored. However, the nature of graphic organizers is to outline or abbreviate rather than to give supporting information and/or explanations that are usually required by the questions.

MYTH: Doing more than required by the prompt will score a 4.

MYTH BUSTER: A score of 4 requires the response to completely and accurately reflect the correct answer according to the scoring guide. Extra information is not required to score a 4. If the extra information
is inaccurate, it can prevent the responses from scoring a 4.

MYTH: Content-specific vocabulary must be used in order to score a 4.

MYTH BUSTER: Not necessarily; if the content can be adequately expressed without the use of specific vocabulary, appropriate credit will be given to the response. Content vocabulary is only required if the question specifically asks for its use.

MYTH: Three or more examples always must be given.

MYTH BUSTER: No; the question will specify the number of examples required. Giving more will not increase the score.

MYTH: Scorers only have 30 seconds to score each piece.

MYTH BUSTER: No; scorers can take as much time as needed to score each response.

MYTH: Released items are “bad” items that have been thrown out of the test.

MYTH BUSTER:Not true; the items are good to use in instruction and are representative of the test. Release of items is not based on how the
items performed.

MYTH: “Strike-throughs” on an open response are better than erasing.

MYTH BUSTER: Not necessarily; scorers will score what is written on the page. Strike-throughs may take up more space than erasing, leaving less space to complete a proper response.

MYTH: Open response answers will be scored using the new analytical
scoring.

MYTH BUSTERS: Each open response item has a specific scoring guide/rubric. The questions are scored for content and not writing
mechanics.

MYTH: Reading and mathematics scores at the new grades added in 2007 will be counted for only NCLB.

MYTH BUSTERS: For grades 3-8, all reading and mathematics scores will be used for NCLB and CATS calculations.

KDE says they have been, as yet, unable to pinpoint the original source of the PowerPoint.

SOURCE: KDE communications

KDE says ORQ information attributed to Measured Progress is in error.

Last week KSN&C heard complaints from teachers that the process for scoring Open Response Questions (ORQ) was changing in midstream....and just weeks before administration of the CATS test.

Teachers claimed there was a disconnect between the instructions received from the Kentucky Department of Education and those from its testing vendor, Measured Progress.

Now, KDE says it's all a mistake and that they are tracking down and correcting the problem.

Apparently, some Grant County teachers recently went to a Measured Progress conference and brought back (or perhaps created?) a PowerPoint slide show titled "Measured_Progress_Conference[1].ppt." The slide show indicated that following KDE's instructions would actually be counter-productive to students scoring well - an obvious problem. That PowerPoint was distributed to teachers widely.

KDE Communications Director Lisa Gross told KSN&C today,

"We heard about this last week and began to look into it. Measured Progress says that its staff did not create the PowerPoint and doesn't know where it originated. The information in the PowerPoint file is incorrect and false."

On the final slide, the PowerPoint listed the following resources:

Julia Payne –Lewis, Measured Progress jlewis@measuredprogress.org

Marcia Tibbetts, Program Manager Measured Progress Mtibbetts@measuredprogress.org

But, it is not clear that they produced the slides. Gross went on to say,

"We and Measured Progress are trying to track down the source of the document. If we're able to do that, we'll work with the author(s) to correct the misinformation and discuss with the author(s) the mis-citing of sources that weren't involved in its creation."

KDE has agreed to keep KSN&C informed whenever the situation is resolved.

So whatever the situation...

...bad info from Measured Progress
...bad note-taking from conference attendees
...or something else

Teachers complaining about the issue seems to have had the beneficial effect of getting KDE and Measured Progress on the same page...and clarifying for all specifically how students will be scored on ORQs. Further KDE has indicated it will send new guidance to all district testing coordinators to resolve any misunderstandings.

What fresh hell is this? ...CATS Now Penalizing Students for Following Test Directions?

Educators complain of lost time

and bad information from KDE

Some Kentucky teachers are mad.

Not your run-of-the-mill kind of mad, but the kind that comes from feelings of betrayal; like when you boss gives you bad information that only serves to keep you from reaching your goals - the goals your boss will later blame you for missing.

Some think KDE has withheld vital information on the CATS assessment - information about questionable assessment practices. Others have welcomed the news, as though they just received insider information that might give them a competitive edge against other schools.

Kentucky School News and Commentary started hearing concerns from teachers yesterday that the scoring directions being provided to the schools by KDE is in error; and following KDE's directions would prevent students from receiving high scores on their open response questions (ORQ).

As one Kentucky principal explained it,

"...Here is what KDE is not telling you... we sent a team straight to the testing source (Measured Progress) and found out that the way we have been told to do ORQs by KDE will get us no higher than a 2 with this new company. To get higher -- students MUST make connections, inferences, and extensions."

And, the spectre of ethical concerns was raised by the approach being used by Measured Progress. Some Grant County teachers learned that,

"The question does not always mean what it says. There is hidden meaning in some of the questions...List does not just mean list. The hidden meaning is list and explain or list and describe."

This means that Kentucky students (some as young as 9-years old) will be asked to do one thing...but will be assessed on something else - some secret rubric. How is a student to know which questions have a secret rubric, and which do not? What are the ethics of penalizing students for following instructions?

Measured Progress told the Grant Countians that,

"We must coach the students up on this. As a matter of routine, students must do more than what is asked of them by the question prompt."

So, they ask the kids a question, and if a compliant student answers precisely as instructed, they get a weaker score.

This is NOT how Kentucky teachers have been instructed to prepare students or score ORQs. KDE's Tips for designing Open Response Questions directs teachers to,

"specify exactly what you want students to answer... as opposed to ambiguous terms ... Use simple and direct language. You are evaluating what students know rather than attempting to determine if they can decipher the question."

Measured Progress not only requires students to decipher the questions, but to divine them.

This is lousy enough. But if that's the way its going to be, why didn't Kentucky teachers learn about it much earlier, and from KDE?

The National Council on Measurement in Education's Code of Professional Responsibilities in Educational Measurement says,

"Those who prepare individuals to take assessments and those who are directly or indirectly involved in the administration ... including teachers, administrators, and assessment personnel, have an important role in making sure that the assessments are administered in a fair and accurate manner. Persons who prepare others for, and those who administer, assessments have a professional responsibility to...inform the examinees about the assessment prior to its administration, including ...how the assessment information will be judged or scored..."

Well, it's now mid February (weeks away from the CATS testing window) and Kentucky teachers are just now learning how they must prepare their students to score well - and not from KDE.

NCME's ethical code also states,

"Persons who educate or inform others about assessment have a professional responsibility to...remain competent and current in the areas in which they teach and reflect that in their instruction..."

"provide reasonable opportunities for individuals to ask questions about the assessment procedures or directions prior to and at prescribed times during the administration of the assessment." ...

Kentucky teachers have apparently had some opportunity to ask questions, but have been given wrong answers by the folks in charge.

Did KDE know about this...but not say?

Or, did KDE not know what Measured Progress was up to?