Setting performance standards requires a great deal of subjective input from judges. While no method appears to be superior they all rely on the subjective decisions. Identifying and selecting quality judges is extremely important. Judges must be qualified to make judgments on the test (AERA, APA, NCME, 1999). The purpose of this review is to examine the criteria for selecting judges and the common training methods used to produce correct performance standards. Selecting the best judges impacts the amount of training needed to set a successful standard. The quality of the judges is inversely proportional to the amount of training required (e.g., better judges, less training). Raymond and Reid (2001) propose six criteria or characteristics of qualified judges. The judges should be: (a) content matter experts; (b) knowledgeable about the examinee population; (c) able to estimate item difficulties; (d) aware of the opportunity to learn (environment); (e) appreciative of the ramifications of setting the standards; and (f) sure all relevant interests are represented in the establishment of the standards. The number of judges should be large enough (n=10-15) to ensure accurate results. The following questions have been used to evaluate the judges¢ performance: Do the judges¢ ratings remain stable over time? Are they consistent with the tasks of the method employed, such as rating the minimally acceptable candidate¢s performance? Is there a realistic expectation of examinees¢ performance using the standards (% pass and fail)? To ensure that all relevant interests are represented and the remaining criteria are reached, training of novice judges is needed along with educating judges who may be deficient in areas like content matter experts in more global issues outside their expertise. Exposing the judges to the test-development process focuses on the test¢s purpose and the consequences of the standards for the examinees. Practice in the tasks required, such as estimating proportionally correct values and developing a set of knowledge, skills and abilities needed for both judges and the examinees are important tools. Like any learning situation, feedback is very powerful and necessary in training the judges. These guidelines are general and can be used with many of the performance-standard methods and on many different types of tests from general tests to credentialing tests. Regardless of which method is used, some selection and training process will be needed. The resources, personnel and type of test will dictate the amount and type of training needed.