Grading the WASL -- Part 1

Aired Thursday, May 25, 2006

By Liam Moriarty

Email this story to a friend

Anchor Lead: More than two dozen states - including Washington - have made passing a standardized test like the WASL a requirement for graduation. But do students always get the score they deserve? The company that scores the tests says yes. But some former test graders are not so sure. KPLU's Liam Moriarty has the story. 5:02

Could the kind of confusion scorers report on the Ohio Proficiency Test happen with the WASL. Tomorrow, in Part 2, Liam talks to testing experts and Washington Education Officials to find out.

Listen Now! MP3

Full Story Text:

About three years ago, John Koudela was between jobs. The long-time Burien resident had been laid off by a tech company after Seattle's dot-com bust. So he was excited when he saw a newspaper ad for a temporary job grading standardized tests for NCS Pearson, the nation's top scoring company.

KOUDELA: It required a college degree, and some simple tests to take and an interview, and I thought, yeah, I can do this.

Anna Rhodes heard about the job from a friend who'd already been hired. She recalls the company seemed eager to hire a lot of people, fast.

RHODES: It was sort of walk in and out in half an hour and you've go the job. I've never heard of any other job where that's true. Even at temp agencies they've interviewed me for longer.

Like John Koudela, Rhodes had been doing the temp job shuffle, and welcomed the chance to make $11.40 an hour for a six week stint at Pearson's testing center in Auburn. After a half day of training and practice, Koudela and Rhodes joined several hundred other temp workers grading the Ohio Proficiency Test. That's Ohio's equivalent of the Washington Assessment of Student Learning, or WASL. Workers in groups of a dozen or more would sit at computers and bring up scanned images of student's test papers.

RHODES: And the idea was to quickly assign it a score and move on to the next one that popped up on your screen.

For open-ended or essay-type questions, workers were given a scoring guide -- called a rubric. They'd use the rubric to score student's answers against the range of answers deemed correct. Rhodes - who has no science background -quickly ran into problems scoring an elementary grade science test. She and others kept coming across answers that looked correct, but were not on the rubric.

RHODES: I would ask the team leader, I would ask the people on either side of me, y'know, "Have you gotten this one before, anything like this? How did you score it?" And over lunch I would actually call my dad because he's a physicist.

Rhodes says supervisors would sometimes clarify which answers would get credit and how much. Other times, they asked for a ruling from a committee of testing experts … Meanwhile, John Koudela's group was having similar trouble. He says handwritten sheets of additions, modifications and clarifications were posted all over the cubicle walls, so scorers could keep track of the new answers.

KOUDELA: The rubric got bigger and bigger and bigger each day.

One science question was so troublesome it generated 27 pages of annotations.

KOUDELA: I have to keep looking at my notes, and say, yeah, OK, he's talking about land erosion. (turning pages) OK, let's see, land erosion. That might be over here ... No, it's not over here, it's on this one.

That took a toll on his effectiveness - and his morale.

KOUDELA: I got to the point really where I knew my scoring accuracy was going down, because my lack of confidence to be able to score every single test question, it was like I had to double-check every single time.

To Koudela and Rhodes, the whole process seemed terribly haphazard … And - with a constantly-changing scoring guide -- the potential for unfair grading seemed huge. But the scoring company says not to worry.

VICKERS: I would just say first of all, if there were 27 pages of notes it wasn't haphazard. I would say someone was putting a great deal of thought into it.

That's Daisy Vickers. She's director of performance scoring for Pearson. Pearson scores tests for Ohio and nearly two dozen other states, including Washington. Vickers acknowledges even the most-thoroughly-prepared test questions can't always anticipate how students will respond.

VICKERS: If new things kept coming up as the scoring progressed, it would only be logical to have more decisions to deal with those.

Decisions about adding to the rubric are made by consulting the committee that helped write the scoring guide - and state Department of Education officials. If there were any major inconsistencies in the Ohio test, Vickers says, she'd know about it. She adds that the temporary employees doing the scoring may not be aware of all the precautions being taken behind the scenes to make sure the tests are accurately graded. But John Koudela isn't convinced ...

KOUDELA: The nut of the problem is that we have a large group of students that are answering these questions for which not even the scorers know what the answers are untin the tests are begun to be scored.

Pearson scores 40 million tests each year, and enjoys a good reputation in the testing field. But over the years, the company has also gotten a number of high-profile black eyes. Mistakes in Minnesota, Michigan and other states have produced thousands of incorrect test scores. Six years ago in Washington, the company gave artificially high grades to more than 400,000 essays on the WASL. Pearson says each incident has resulted in improved safeguards ... The company has elaborate checks and balances to make sure tests are scored consistently, and Daisy Vickers is confident they work. But the ultimately, she says, it's up to the customer - the state that's paying Pearson - to set the standards the company uses to score their students' tests.

Liam Moriarty, KPLU News