Scheduled for Measurement Symposium - New Perspective and Practice in Setting Performance Standards, Tuesday, April 1, 2003, 3:00 PM - 5:00 PM, Convention Center: 113C


Setting Performance Standards Using the Bookmark Procedure

Minsoo Kang and Weimo Zhu, University of Illinois at Urbana-Champaign, Urbana, IL

While the modified Angoff method has been widely used in practice, it has several limitations, e.g., labor intensive and difficulty in producing accurate estimates (Shepard et al., 1993). To address the limitations, research scientists at CTT/McGraw-Hill have developed a new method called the Bookmark method (Lewis et al., 1996), which requires the following steps: (a) After ordering items/tasks from easiest to hardest, which creates ordered item booklets, a group of judges reviews the ordered booklets and discusses the skills and knowledge of the each item/task in small groups; (b) Each judge places a bookmark among the items/tasks representing a cut point (e.g., the threshold of a performance level); (c) The small groups then systematically review and discuss each judge’s ratings of the ordered booklets; (d) The judges in the small groups make second ratings after the review and discussion and, using the information in this second-round of rating, the means or medians of the cut-off scores are calculated by group; and (e) The rating results from all the groups are presented to the entire group for the final judgment of the cut-off scores. Developing performance level descriptors will also be included in this step. Mitzel, Lewis, and Green (2001) illustrated several advantages of the Bookmark procedure, including: (a) Integrating different item formats into a single mixed format, which makes it possible to place two item types onto a single scale, (b) Simplifying the judgment task by reducing the cognitive load required by the modified Angoff method, (c) Connecting the judgment task of setting cut-off scores to the measurement model, and (d) Connecting test content with performance-level descriptors, which is important in order to inform students, parents, and teachers what content students must know and be able to do to achieve a given performance level. This presentation will provide a step-by-step description of the Bookmark method. Also, its application potentials in physical education measurement practices will be discussed.

Back to the 2003 AAHPERD National Convention and Exposition