Governor Steve Beshear has signalled Republicans that KERA is on the table for discussion - and that sentiment has been echoed by no less than Harry Moberley, who single-handedly killed Senate Bill 1 the last time around - it's no longer reasonable for KERA fans to assume that education reform in Kentucky will remain the same after the current legislative session. It's probably a good idea to think seriously about what those changes might bring.It is also important to look at the interaction of several simultaneous efforts. For example, everyone is focused on budget cutting and further reductions to KDE are in the cross hairs of some legislators. At the same time SB 1 intends to replace assessment in several areas with "program evaluations."
Now, the last time I talked to Elaine Farris about her already decimated KDE staff, I think she indicated the curriculum department was down to one or two language arts consultants. Further reductions to KDE could leave ...what? ... none? Nobody in their right mind expects a program evaluation to have the same motivating power as an assessment. The department's ability to drive the program evaluation at the local level is something close to zero. So it will fall to each district to police themselves.
Sweet.
If SB 1's intention is to take the pressure off schools to perform for all students, it's on the right track. But, I think we can predict weaker student performance as a result.
Here are some random musings on the new, but not necessarily improved, Senate Bill 1. Each bullet is a proposed change followed by my questions or comments.
- "add an expectation relating to performing arts" OK. Fine. Objectives that help students become better public speakers would be particularly welcome. But apparently these objectives won't be assessed, so principals across Kentucky will know that any efforts to meet the requirement will only have to be superficial to pass muster.
- "restate the purposes and components of a balanced assessment program including both formative and summative assessments" This could be good. In fact, if the original KERA deliberations had been given time to build an accountability system properly, it would have started with curriculum standards; formative assessment; adjusted instruction; summative assessment; and the accountability system on top of that. Instead it was implemented in a top-down fashion, which contributed to its unpopularity.
- "eliminate the open-response questions requirement" This is a wrong idea. Proponents of SB 1 have elevated legitimate concerns over reliability to heights that may ultimately hurt students. Students must be strongly encouraged to think and write. Open response items, however imperfect, require higher-order thinking on the part of students. Any retreat from that goal is problematic.
- "require writing portfolios be maintained for each student in grade 5 to 12 but eliminate from the state assessment" Neutering the writing portfolios will cripple writing instruction in many places. Whatever remains in the assessment will draw the focus of teachers and principals.
- "limit rewrites of the writing portfolio" I don't even know what this means? Limit rewrites? What constitutes a rewrite? And who's going to monitor this...and how? If a student corrects a capitalization error, is that a rewrite? What if she changes two words? ...or a paragraph?
- "require an on-demand assessment of student writing one time within the elementary, middle, and high school grades, respectively" More of less.
- "require writing assessments consisting of multiple-choice items emphasizing mechanics and editing one time within the elementary, middle, and high school, respectively" Arguably the dumbest part of the bill; SB 1 seeks to test student writing without the students actually having to write. A more reliable but less valid test is not progress.
- "require each school council to develop policies relating to the school's writing program" So what?
- "eliminate practical living and vocational studies from the assessment program but require a program evaluation of practical living and career studies annually" Death to that part of the curriculum. The program evaluation is a waste of time and money.
- "eliminate arts and humanities student testing from the assessment program but require a program evaluation of arts and humanities annually" See above.
- "require that accelerated learning be provided any student whose scores on any of the assessments indicate skill deficiencies" and "require each school to devise an accelerated learning plan" As if teachers are not already tearing their hair out trying to get some students up to proficiency.
- "require individual reports to parents on the achievement of their children compared to school, state, and national results" This is the heart of SB 1. The most difficult part of KERA for many parents was its refocusing of student assessment from individual performance to school performance. There was a good reason to do it, but it was never well-accepted by many parents. Parents want to know how their children's performance stacks up against other kids the same age. Better, is to keep standards high while building a value-added system that includes as much data as we have; but which resists the temptation to use that data inappropriately and punitively against teachers.
- "limit state core content testing to the last seven days of a local district's school calendar and limit number of days of testing to no more than five during that period" Limiting the time required for formal assessment may be a good idea. You don't fatten a calf by weighing it. But if I had a magic wand, Kentucky would conduct three days of normative test in the fall of the year and keep a week of state assessment in the spring.
- "provide that a local board of education may adopt the use of commercial assessments for making formative judgments" Sure.
- "require the Kentucky Department of Education to assist districts in selection of commercial products that address Kentucky core content" Sure, if KDE has any assessment folks left after budget cuts.
- "require the Kentucky Board of Education to recommend at least three companies or products to the state board for approval" OK.
- "require the School Curriculum, Assessment, and Accountability Council to provide recommendations relating to the identification of academic skills and deficiencies of individual students" OK.
- "modify language relating to successful school and delete references to rewards" Rewards, like other free-market ideas have lost their luster.
- "modify title of the assessment and accountability system, by adding reference to "academic achievement" CATS is seen as a bad brand, so fine, rename it.
- "change the high school readiness exam from grade 8 to grade 9" If there's a reason to support or oppose this, I don't know what it is.
- "change dates for reporting student data to each school council and change dates for actions by school councils" OK.
So what does all of this mean? It means that when Bob Sexton says, "It's more important to do it right than to do it fast," he's not just dragging his feet - he's right. Changes to KERA need to be considered holistically, carefully and without inviting unanticipated consequences due to a rush to "do something."
2 comments:
Richard,
I think you have a number of conceptual problems with SB 1 and what is going on in our schools. You also nail some issues. Here are a few of your comments offset in >>> <<< and my thoughts.
>>>"add an expectation relating to performing arts" OK. Fine. Objectives that help students become better public speakers would be particularly welcome. But apparently these objectives won't be assessed, so principals across Kentucky will know that any efforts to meet the requirement will only have to be superficial to pass muster.<<<
We’ve struggled with the highly subjective area of assessing the arts since day one of KIRIS. Nothing works adequately, and the present multiple-choice loaded assessment is very unsatisfactory, tending to reduce school interest in real arts efforts like band, glee club, and so forth. The replacement arts assessment idea discussed in the Assessment and Accountability Task Force has very little research behind it (it’s only been experimented with in elementary schools) and will be terribly subjective as well. Requiring an audit of a school-developed program to insure that the arts are in the curriculum looks like the most reasonable way to go at this time.
>>>"restate the purposes and components of a balanced assessment program including both formative and summative assessments" This could be good. In fact, if the original KERA deliberations had been given time to build an accountability system properly, it would have started with curriculum standards; formative assessment; adjusted instruction; summative assessment; and the accountability system on top of that. Instead it was implemented in a top-down fashion, which contributed to its unpopularity.<<<
I am glad you now agree with me. I have been saying this since 1994 when I first started studying our assessment program. We did indeed create our assessments backwards, an elemental mistake that helps to explain some of our current problems.
>>>"eliminate the open-response questions requirement" This is a wrong idea. Proponents of SB 1 have elevated legitimate concerns over reliability to heights that may ultimately hurt students. Students must be strongly encouraged to think and write. Open response items, however imperfect, require higher-order thinking on the part of students. Any retreat from that goal is problematic.<<<
You are 20 years behind the times with your out of date assertion that only open-response questions can tap higher order thinking skills. For example, the recent report of the National Mathematics Advisory Panel explicitly says “The evidence in the scientific literature does not support the assumption that a constructed-response format, particularly the short-answer type, measures different aspects of mathematics competency in comparison with the multiple-choice format.” (Page 60)
However, that isn’t really the most important issue. Even if we assume that open-response questions can tap something that modern, well-designed multiple-choice questions miss, in the CATS system, who winds up with that knowledge? It isn’t the school or the student’s teacher.
Schools don’t get much feedback from CATS – mostly just a score. The schools don’t get annotated student answers back from the graders. In fact, it is even illegal for schools to keep copies of the original questions.
So, only the grader might have some real idea about what a student can, or cannot do. However, that knowledge never is effectively transmitted back to the school. Of course, this assumes the grader – who might have as little background as just 60 hours of college credit – is really qualified to make such determinations with any degree of informed accuracy. Imagine that, someone with only 60 college credits telling our teachers, who mostly hold Master’s Degrees, how well their students perform. Yeah, right!
So, even if there is some extra knowledge being developed from the CATS open-response questions, that knowledge isn’t being fed back to the schools in an effective and useful way.
This is the big fallacy in the idea that open-response questions on CATS somehow improve our knowledge of how well students think. The CATS cannot give us that benefit despite all the costs and score return delays that come from using these imprecise assessment formats.
>>>"require writing portfolios be maintained for each student in grade 5 to 12 but eliminate from the state assessment" Neutering the writing portfolios will cripple writing instruction in many places. Whatever remains in the assessment will draw the focus of teachers and principals.<<<
You make another badly-out-of-focus assertion. During the Assessment and Accountability Task Force Doris Redfield – the only testing expert on the panel – did a check on writing portfolio use in other states. No other state now uses them for accountability – not one. The few that once did have abandoned them for this purpose. That doesn’t mean that writing portfolios are a bad instructional tool, far from it. It’s just that putting portfolios into accountability inevitably creates a lot of awkward and burdensome rules that teachers have to follow while correcting the portfolios. That just ties our teachers’ hands behind their backs.
What is being done elsewhere? Tennessee is a great example. They do on-demand writing assessment once at each school level and also test mechanics and spelling with a multiple-choice test (exactly like SB 1 would require). It works. Despite demographics that, if anything, would favor Kentucky in any comparison (We have a much smaller African-American population and exclude more of our learning disabled kids), Tennessee’s proficiency rates on the 2007 NAEP Grade 8 Writing Assessment were uniformly higher than ours – for the overall average score, for Whites, for Blacks, for students in the free and reduced cost lunch program – and, for learning disabled kids, Tennessee outscored our proficiency rate by nearly four times. And, that was after Kentucky excluded a far larger proportion of its learning disabled students.
>>>"limit rewrites of the writing portfolio" I don't even know what this means? Limit rewrites? What constitutes a rewrite? And who's going to monitor this...and how? If a student corrects a capitalization error, is that a rewrite? What if she changes two words? ...or a paragraph?<<<
Keep in mind, this is legislation, not an implementing regulation. Clearly the legislators want to cut down on well-known problem of excessive rewriting of portfolios. However, as you properly imply, this is going to require more precise discussions best left to technical experts who craft the supporting regulation.
>>>"require an on-demand assessment of student writing one time within the elementary, middle, and high school grades, respectively" More of less.<<<
You lose me. What do you mean by “more of less”?
>>>"require writing assessments consisting of multiple-choice items emphasizing mechanics and editing one time within the elementary, middle, and high school, respectively" Arguably the dumbest part of the bill; SB 1 seeks to test student writing without the students actually having to write. A more reliable but less valid test is not progress.<<<
First of all, the bill clearly keeps on-demand writing in the revised assessment, so your allegation that students won’t have to write anymore is simply wrong. However, concerning the additional multiple-choice testing of mechanics, this apparently already works down in Tennessee, and elsewhere such as in the ACT. Now, saying something is dumb when it’s already providing value elsewhere….
As far as the validity of the writing portfolios in CATS, you obviously are unaware of the statistics in the past three years’ audits of the program. Not only are there continuing major problems with scoring (even Ken Draut at KDE admitted that to the Assessment and Accountability Task Force), but scoring accuracy is deteriorating in both elementary and middle school portfolio scoring.
>>>"require each school council to develop policies relating to the school's writing program" So what?<<<
Apparently not all councils do this. But, the real reason for this language is to provide some teeth when a writing review team comes from Frankfort. There is no question that these policies must exist in writing so the audit team can both evaluate the policies and make sure they are being implemented. This is just good legislative writing.
>>>"eliminate practical living and vocational studies from the assessment program but require a program evaluation of practical living and career studies annually" Death to that part of the curriculum. The program evaluation is a waste of time and money.<<<
Do you know what happens when a school gets low CATS scores? The answer is that a team is sent in to do a program evaluation. Is that a waste of time and money?
No, most people know, if company is coming, you clean house. If you have a lot of unexpected visitors, you always keep your house in shape.
Anyway, if this does really adversely impact the arts, we can always add this element back into the assessment again. Plenty of teachers say the current assessment is actually a detriment to true arts programs, so why are you so averse to trying the program evaluation approach?
"require that accelerated learning be provided any student whose scores on any of the assessments indicate skill deficiencies" and "require each school to devise an accelerated learning plan" As if teachers are not already tearing their hair out trying to get some students up to proficiency.
Teachers are tearing their hair out, but is the current effort being effectively pursued? This is actually like the previous item about requiring writing program policies to be in writing. You need this for the audit team to have handholds when they come in to evaluate the program.
>>>"require individual reports to parents on the achievement of their children compared to school, state, and national results" This is the heart of SB 1. The most difficult part of KERA for many parents was its refocusing of student assessment from individual performance to school performance. There was a good reason to do it, but it was never well-accepted by many parents. Parents want to know how their children's performance stacks up against other kids the same age. Better, is to keep standards high while building a value-added system that includes as much data as we have; but which resists the temptation to use that data inappropriately and punitively against teachers.<<<
This is indeed a major change in SB 1, and a good one. We spend far too much to get back far too little from CATS.
The individual CATS student scores are not valid and reliable due to the matrixing format, and that format in turn is driven by the continued infatuation of some with open-response questions. Because you can’t ask each student enough open-response questions in a reasonable amount of testing time, we wind up having to matrix in order to cover the material. But, that means each student is only tested on a portion of the curriculum each year. That in turn means you cannot do valid and reliable longitudinal analysis of individual students because they randomly get tested on different subsets of the curriculum each year. And, on it goes. The fundamental design of CATS is hostile to valid and reliable open-response testing. Sometimes, life just works out that way.
>>>"require the Kentucky Department of Education to assist districts in selection of commercial products that address Kentucky core content" Sure, if KDE has any assessment folks left after budget cuts.<<<
This is one reason why we need to make our changes to CATS now. If this element of the bill passes, then KDE will have to make its cuts in other areas that won’t need so much staffing. The legislature needs to tell KDE what it wants soon so the wrong people don’t get cut.
I'm pretty sure I'm familiar with what's going on in the schools, since between the two of us, I'm the one who's actually been there. And I do have some problems with SB 1, not the least of which is its hodge podge approach. This is the wrong way to go about it.
If the legislature wants a different kind of system, they should direct the board of education to create it and avoid getting involved in the details.
But I am glad that you agree with me on some points.
Post a Comment