Beshear to Appoint New Board Members to Implement Senate Bill 1

Over at Dr H's Blog, Education Commissioner Terry Holliday reminds us that Governor Steve Beshear will soon replace as many as seven members of the Kentucky Board of Education.

Those whose terms expired on the 14th of this month (and who typically serve until replaced) are: Chairman Joe Brothers, Vice Chair C.B. Akins and members Kaye Baird, Jeanne Huber Ferguson, Austin W. Moss, Judy H. Gibbons and Doug Hubbard.

That leaves Vice Chair Dorie Combs, Billy Harper, Brigitte B. Ramsey and David K. Karem whose terms expire 4/14/2012.

The new board will have the task of implementing Senate Bill 1 which calls for a new accountability system, internationally benchmarked standards, formative and diagnostic assessments, and thankfully, student growth measures.

Vision for Education Reform in Kentucky

The Commish says,

I have started the process of engaging all advisory councils in the dialogue concerning a new accountability system and strategic plan for the Kentucky Board of Education. The board will hold a strategic retreat prior to its June meeting... The key to our success in transforming education in Kentucky will reside in our ability to focus on a few goals with a few strategies that are done with precision and fidelity. These few goals will focus on the vision of every child proficient and prepared for success.

This vision will be measured by indicators of proficiency, growth and closing gaps among student groups.

Proficiency will be measured by the cohort graduation rate and our comparative position among states.

Growth will be measured by the increases in our annual percentage of high school graduates who are prepared for college and career as compared against other states.

Closing gaps will be measured by the decreases in gaps for the graduation and readiness rates among student groups in Kentucky as compared against other states.

Holliday sees the National Assessment of Educational Progress (NAEP) along with the Educational Planning and Assessment System (EPAS) system as the "ultimate measures" of Proficiency, Growth and Gap rates.

The NAEP is the only nationally representative longitudinal assessment of what America's students know and can do in various subject areas. Assessments are conducted periodically in mathematics, reading, science, writing, the arts, civics, economics, geography, and U.S. history.
Since NAEP assessments are administered uniformly using the same sets of test booklets across the nation, NAEP results serve as a common metric for all states and selected urban districts. The NAEP has been used to show how some states lowered their own standards under the pressure of NCLB's current accountability model; a model Education Secretary Arne Duncan says he wants to change.

The EPAS system is required in Kentucky but does not enjoy NAEP's nearly universal acceptance due to its contrived set of benchmarks. Testing experts, like Skip Kifer, say the Hierarchical Linear Modeling EPAS uses is statistically indefensible and creates flawed benckmarks because it ignores institutions. Students are part of institutions and should be modeled that way.

Holliday's strategic priorities are based on federal guidance provided by Race to the Top, State Fiscal Stabilization Funds, the proposed plans for the reauthorization of Elementary and Secondary Education Act (NCLB), SB 1, SB 168, SB 130 and other state statutes.

The new lingo is:

• Next Generation Learners
• Next Generation Professionals
• Next Generation Support Systems
• Next Generation Schools and Districts

In addition, Governor Steve Beshear has appointed a Transforming Education in Kentucky (TEK) Task Force which is scheduled to make recommendations for the 2011 legislative session.

Holliday has called for comments and suggestions from the public and the alphabet groups. The TEK Task Force will conduct community forums and focus groups in August to gather input.

Senate Bill 1 ends the CATS era. What will replace it?

By Skip Kifer

SB1 proposes a huge amount of testing. Some testing is statewide; other testing is not. Some tests are used for accountability purposes; others are not. Some tests are multiple- choice only; others are not. There are formative tests, summative tests, performance assessments, benchmark tests, interim tests, norm-referenced tests, criterion-referenced tests, end of course tests, and writing portfolios as well as program reviews and program audits which are used to measure students, schools and districts.

The Kentucky Department of Education (KDE) is directed to integrate the testing into a new assessment system. It has almost three years to complete the task.

I doubt that a defensible new system can be created, even with that amount of time, without additional resources for KDE and more focused testing.

A first step to building a new assessment is to find out whether what is presently being done works. Major components of the existing system should be evaluated. For instance, EPAs - Explore, Plan and ACT - produces results presumed to help students learn and school personnel make better decisions. Does it? What is the evidence that the information is used? What is the evidence that using the evidence makes a difference? On what? Studies of each major component should provide answers to those or similar questions. If the component is not working, it should be eliminated.

There should be a thorough understanding of the implications of moving from an assessment system focused on schools to one focused on students. CATS relied on different forms of the test to get a better sample of achievement within a school. A student might spend an hour taking a test but with six forms of the test, the assessment contained about six times as much information for each school. Even with that, about 25% of schools were misclassified. Some were said to be progressing when they were not. Others were said to be needing assistance when they were progressing.

The existing assessment had thousands of misclassifications at the student level, too. Proposing to produce longitudinal data on students places more pressure on the assessment and more obstacles to accurate measurement. Creating measures that can be scaled across grades suggests spending more time testing each student. Is that desirable? Is it practicable?

Finally, the biggie! After almost two decades of educational reform with a spotlight on accountability through high-stakes testing, is their evidence that such testing works?

Research results both in Kentucky and nationally are mixed. Test scores go up but there are questions of whether that is mainly because of teaching to the test. Because what is taught and tested narrows, higher test scores may not mean mastery of a content area and may mean that content areas not tested are not emphasized. In addition, states with high-stakes testing do no better, and perhaps worse, on national tests than states without such programs. Reasonable persons would agree that test scores are, at best, a narrow reflection of successful schools. Other aspects are more important. Evidence should be collected, therefore, that indicates whether after about two decades of high-stakes testing Kentucky schools:

a) are better places for kids than they were prior to the reform;
b) nurture talent in ways that it should be nurtured; and
c) insure those who emerge from the school system have commitments to democratic ideals, participation in democratic communities, and tolerance for varied persons and views.

Armed with good evidence, resources and time, I hope KDE creates a system that benefits the Commonwealth and its children.

Kifer says EXPLORE and ACT Overstate Claims

Skip Kifer of the Center for Advanced Study of Assessment at Georgetown College released a report this month that calls into question claims made by the ACT program regarding the efficacy of its EPAS system.

Kifer has a history with the ACT... and its a good one. In fact, he has to have been considered a fan. In Mental Measures Yearbook (Kifer, 1985) he lavishly praised test-maker ETS and the technical construction of the ACT. In fact, he found their practices to be worthy of admiration and emulation.

But times change. Since states have begun requiring exams that are specific to state curriculum, ETS has entered the state market and made claims about the efficacy of the EPAS system. It is these claims Kifer examines.

The study was motivated by the introduction of Senate Bill 1 during the 2008 legislative session. The bill included additional reliance on off-the-shelf standardized tests. Since the state assessment already contains such measurements under the EPAS system, a relatively new ACT program, Kifer set out to determine whether the program lived up to its claims.

Kifer looked at the characteristics of EXPLORE and ACT, two components of EPAS taken by 8th, 10th and 11th or 12th grade students, to determine whether or not the claims made for it and them are valid. He also compared ACT predictions to those obtained from other components of the Commonwealth’s assessment.

Kifer concludes that claims for the efficacy of the EPAS are stronger than the findings would suggest.

For example, EXPLORE lacks a representative national comparison group and therefore lacks true national norms. EXPLORE was constructed on the basis of national information, and therefore does not represent the Kentucky curriculum. Kifer also says,

"Although the technical manual suggests that the tests measure 'higher order thinking skills,' there is no dimension for those attributes in the test construction descriptions and no indications of how many questions of that type are on a test. EXPLORE is alleged to be diagnostic but is diagnostic in the limited sense that scores on EXPLORE predict scores on the ACT. Finally, there are validity studies of Kentucky’s assessments and ACT scores that suggest that the Commonwealth’s assessments are comparably effective predictors of performance in higher education."

Kifer concludes that EXPLORE and ACT do not live up to their claims. It also shows that other measures in the Commonwealth’s assessment do about as well and sometimes better than the ACT tests in predicting grade point average in college.

What this means, says Kifer is that,

If a student were certain that he or she was going to attend a public university in Kentucky, it would not be necessary to take the ACT. That student’s CATS score would be sufficiently predictive to take the place of the ACT. A consequence of using CATS scores in a college admissions process might motivate high school students to do their best on the statewide assessments. Finally, not requiring the ACT could save the Commonwealth a substantial amount of money.