A modest Proposal on Writing Portfolios: Keep Them. Make Them Pass/Fail

So far the suggestions regarding what to do with writing portfolios in the CATS assessment have been polarized. In or out.

Proponents fear (and teachers know) that removing writing portfolios from the assessment will take the instructional focus off this important discipline.

Opponents argue that subjective nature of writing lowers the reliability of the assessment to unacceptable levels.

In my opinion, both are correct.

So what to do?

We can't get rid of the instruction our students need to compete globally, but an increasing number of teachers want to do just that. We can't rely on portfolio scoring because it doesn't discriminate very well between what constitutes good writing and what constitutes very good writing.

But portfolio scoring does discriminate between good writing and bad writing.

So let's change how the portfolios are scored. Let's make the Proficient and Distinguished portfolios = PASS. The Novice and Apprentice portfolios = FAIL.

Then, KDE should recalculate how that factors into the overall accountability system.

Why not?

A Lesson on Writing Portfolios for Legislators

The following is the text of a letter sent on Tuesday to state Sen. Dan Kelly, one of the chief opponents of the writing portfolio component of the CATS testing program. The writer is Ricki Rosenberg, a former Cassidy teacher who now teaches at Picadome Elementary School in Lexington...in the Courier-Journal:

Nice job, Ricki.

...I am well aware that many teachers in Kentucky would like to see portfolios disappear. Teaching writing is hard work and teaching fourth-grade writing is even harder. But does that mean we should get rid of it? Of course not! It just means we have to train fourth-grade teachers better. We need to find fourth-grade teachers who are passionate about writing instruction and who believe in its value. Having trained writing coaches in elementary schools will give the entire staff more confidence and ability to teach writing, which will then create a quality schoolwide writing program.

Over the past 15 years, several changes have been made to the Writing Portfolio. First, we used to require six pieces from fourth-grade students; now we require three. If the students come to fourth grade with a background in writing, and writing is taught with enthusiasm each day, then completing Writing Portfolios will not be too strenuous for both students and teachers. This is a realistic, manageable and appropriate assessment.

As time goes on, I believe that our writing instruction has improved and our expectations have increased. ...If a subject is not assessed, it will not be taught as thoroughly and with the same rigor.

With less commitment to writing, students will also have less chance for introspection, analysis and synthesis, all skills to promote higher-order thinking. ... Doing a 90-minute on-demand prompt does not allow for true introspection, analysis and peer review.

As adults, we often write something and let it sit for a day or two before revising it and also share it with a peer to receive feedback. This cannot happen for a child during the on-demand portion of the test, thus making it less like a real-world experience.

Some educators believe that if we remove the Writing Portfolio and increase the weight of the on-demand portion of our assessment, teachers will still teach writing rigorously. However, on-demand only addresses the writing of articles and persuasive letters. I am certain we will eventually see an end to personal narratives, memoirs, short stories and poetry. This would be a real tragedy for our students.

In addition, on-demand is a contrived prompt that is not authentic to students' real lives. ...Student choice is essential for motivating children to have a true, authentic reason for writing.

... I hope you consider my request to keep the Writing Portfolio so that Kentucky children will continue to think at higher levels and proceed forward instead of remaining at status quo.

Task Force on Assessment and Accountability

At last week's CATS Task Force meeting included a PowerPoint from consultant Doris Redfield who looked at Writing Portfolios over time. Various recommendations were presented. Here are some bits and pieces.


Robert Holland (2007): Problems with large-scale portfolio assessments:

• Failure to yield reliable comparative data
• Large differences in the way teachers implement portfolios
• Differences in the opportunities given to students to revise their work
• Variations in the degree of assistance that students receive from peers, parents, teachers, and other sources

Elizabeth Dick, 2008 (writing and literacy expert)

• Maintain the writing portfolio assessment as part of the Commonwealth Accountability Testing System (CATS).
• Emphasize the need for district and school leadership to ensure that the writing portfolio procedures outlined in 703 KAR 5:010 and in Part 1 of the Kentucky Writing Handbook are being implemented and that teachers’ professional growth plans and performance evaluation documents address appropriate expectations for writing instruction. (LS)
• Consider ways to hold students more accountable for their writing performance (HS end-of-course? Graduation? In-class grading practices?)
• Continue to support increased state funding of the eight Kentucky Writing Project sites which provide teaches across grades and content areas with quality professional development through the month-long Summer Institutes, Saturday Mini-Conferences, and Advanced Institutes. (I)
• Consider a course in writing instruction as a condition for teacher licensing in Kentucky. (I)
• Provide writing cluster leaders, who serve as instructional leaders in their schools, with more instructional support and professional development opportunities. (I)
• Provide schools with more specific writing portfolio audit feedback and more tools for ensuring accurate portfolio scoring. (I; could be an assessment issue if portfolio writing remains in the accountability system).
• Schools must be encouraged to analyze students’ working folders across grade levels and accountability grade portfolios annually. (Program evaluation issue?)

Kentucky Association for Assessment Coordinators (2008)

• Any assessment that limits opportunities for learning is a mistake.
• We believe that, if educators focus on teaching for learning, the accountability systems and test scores will take care of themselves.
• Issues of greatest importance include:

• Resolve the writing portfolio issues (see KAAC’s 2004 position paper and
  recommendations)

  Expand the use of on demand writing (at grades 3 through 12 will increase
  more effective writing opportunities – discuss Elizabeth Dick’s point).

Thinking about the CATS Writing Portfolio and the ACT

At last week's CATS Task Force meeting there was a suggestion that Kentucky - being the only state to use writing portfolios in its state assessment - is surely out-of-step with other, more enlightened states.

After all, if writing portfolios were such a good idea, everybody would want them. Right?

Supposedly to fix this problem, those who want to discontinue the problematic CATS suggest substituting the ACT, with its over-advertized benchmarks.

But lo and behold, Kentucky is one of just five states that requires the ACT for all high school juniors.

I'm just sayin' ...

The National Writing Project

August 21, 2008 – Kentucky professors, key educators and teachers will partner with researchers from the University at Albany on the next phase of The National Study of Writing Instruction, a multi-state research project designed to identify the ways in which writing is presently being used and taught in the major academic subjects (English, Mathematics, Science and Social Studies) in the middle and high schools grades. Its goal is to create the knowledge base from which to improve writing instruction in ways that better prepare high school graduates to meet the writing demands of the workplace and college.

This year, researchers from the National Study of Writing Instruction are working collaboratively with university-based researchers and key educators in California, Kentucky, Michigan, New York and Texas who will, in turn, gather data about writing instruction in a subset of schools identified as effective in the teaching of writing. The five states were chosen to reflect a range of approaches to statewide assessment and accountability of student writing across the subjects. The research is funded by the Chicago-based Spencer Foundation as well as the National Writing Project.

Kentucky, in particular, has included writing portfolios as a component of its assessment since 1992. Well-known writing researcher and professor, George Hillocks, has noted that “[t]he Kentucky portfolio assessment of writing is an exemplary assessment by many criteria [which] promote authentic writing, rather than the formulaic.” While the portfolio and on-demand assessments for writing are not unique to Kentucky, Kentucky is the only state to sustain the statewide assessment of writing portfolios over such a long period of time, 1992-present providing valuable data to support the state in its efforts.

In his research, Hillocks “believe[s] that the difference in Kentucky lies in the nature of the assessment, the training afforded the teachers, and the guidelines set by the state. Kentucky has more open criteria for good writing, more teacher education through the eight Writing Project sites, strong guides to the teaching of writing in its various state and local handbooks, and an assessment that permits development of writing in a serious way” (2002). Because these assessments promote such valuable instruction, several Kentucky schools will be included in the National Study of Writing Instruction.

John Hagaman of Western Kentucky University, Sally Martin of Eastern Kentucky University, and Lee Ann Hager, Writing Consultant with the Kentucky Department of Education will be leading the Kentucky team...

SOURCE: NWP press release

Audit: CATS test writing scores inflated, critics want test dumped or altered

Here's Mark Hebert's story on the audit of writing portfolios from Wednesday:

(WHAS11) - An audit of last year’s CATS test has found that nearly half the student’s scores on the writing portfolios were inflated. Critics of the CATS test say that’s proof that the whole test needs to be dumped or at the very least, the writing portfolios need to go. But some teachers say that would be a bad idea...

...A state senate bill that would abolish the CATS test has virtually no chance of passing the house and would face a veto by Governor Beshear if it ever got to his desk.

Video here.

Audit shows CATS Assessment Challenges Persist

Martin Cothran of the Family Foundation issued a challenge to the reliability of including writing portfolios as part of the CATS accountability process. In doing so, he points out a persistent problem.

His claim focuses on the soft underbelly of the CATS assessment, an accountability process that relies on reasonable minds to agree on what constitutes good writing. It's certainly not a perfect science.

In a press release posted on his blog, vere loqui, Cothran reports that "closely held" 2005-2006 audit results show Kentucky's frustrations with inter-rater reliability when it comes to assessing writing portfolios.

“The audit seems to suggest,” said Cothran, “that about 75 percent of the portfolios ranked as “distinguished” in 2005-2006 were graded too high, and almost half of portfolios rated “proficient” were given higher grades than they deserved...

The Family Foundation supports the use of portfolios for instructional rather than assessment purposes. They also support Senate Bill 1 which proposes to test writing mechanics without asking students to write.

Cothran's suggestion that the audit reports are "closely held" seems to lack justification. KSN&C asked for and received the current 2006-2007 report from KDE without objection or inquiry within an hour this morning. By law, KDE must audit portfolios annually to monitor the degree of inter-rater reliability and the report is public record.

There is broad agreement that writing is an important skill. But, accurately assessing such a complex skill is challenging at best. It is unlikely that such an assessment would ever produce reliability numbers that rival, say, mathematics exams. It's the nature of the beast.

And, as Cothran accurately points out, most of the difficulty lies in scoring the best portfolios.

KDE spokeswoman Lisa Gross told KSN&C,

As students become more refined writers, it is sometimes more difficult for two scorers to agree completely on their ability levels. The complexity of their work can cause a score to be "on the edge," meaning that another scorer could consider the work to be of slightly higher or slightly lower quality...What the overall audit results tell us is that we need to enhance scorer training and ensure that everyone is comfortable with the analytic model.

The results, and human nature, might also imply that close calls get bumped up. The only way to even approach reliability is through carefully described process.

Since the 2005-2006 audit Cothran refers to, the scoring methodology for writing changed from holistic to analytic scoring. KDE says this was done in order to provide more feedback on student performance at the local level. Each piece within a portfolio was assigned scores at the subdomain levels of content, structure, and conventions. The subdomain scores were summed to provide a raw score for each of the 10,000 portfolios audited.

It should be noted that the new audit did not affect school scores, but provided research data. Audited schools received reports regarding their scoring, but school scores were not changed.

The scoring methodology included 100% double-blind scoring where any discrepant scores were overridden by an independent third read. Quality control measures included "read-behinds" on all readers at a rate of 10%. Scoring supervisors conducted read-behinds of the read-behinds, also at the rate of 10%.

KDE reports the overall percentage of inter-rater reliability with a tolerance of + or - 1; what they call exact or adjacent scoring. The results for 2006-2007: