This paper describes Colorado's attempt to figure out which accommodations to provide to their grade 5 math assessment. I think it provides a very interesting discussion of the way that high stakes tests are created.
The authors wanted to study what kinds of accommodations would allow ELLs to be tested fairly. The study was performed with the best of intentions, but was seriously flawed. In the end, in order to have any measurable effect, the researchers had to throw out most of their data, which didn't show what they hoped it would. As problematic as that methodology is in a scientific study, the conclusion of the paper was most disturbing.
Based on the results of this study (and on common sense), several decisions were made in constructing the grade 5 mathematics assessment of the Colorado Student Assessment Program. First, every attempt was made to avoid unnecessary linguistic complexity. All of the potential test items in the item pool supplied by the state's assessment contractor were reviewed for linguistic features that appear to contribute to text difficulty but were not related to the math content of the item. Most of the items in the assessment item pool were subsequently modified to meet this criterion. In addition, definitions of non-mathematical words were provided, where appropriate, underneath the test item. In no case was the mathematical complexity compromised.
The reason that this is disturbing is that the study showed no effect on test scores of these accommodations. It suggests some of the reality of creating high stakes testing. People creating the tests do their very best to design the tests and study how best to create the tests. But high stakes testing is a very lucrative industry, and the deadlines are non-negotiable, so if the designers are unable to create the test in response to scientific data, they will decide to use common sense rather than delay the test to study the problem some more.
No comments:
Post a Comment