Background/Purpose The purpose of this study was to demonstrate how Monte Carlo methods can be used in a validity study to make empirically-based decisions about sample size (N) for a desired level of power (π), and, to estimate π for a fixed N under a confirmatory factor analytic (CFA) model with model-data conditions commonly encountered in exercise and sport. Two particular model-data conditions, model misspecification and ordinal data, were modeled in this study. Because the purpose was pursued by way of demonstration with the Coaching Efficacy Scale II – High School Teams, related N recommendations were provided.
Method A nine-step procedure proposed by Paxton, Curran, Bollen, Kirby, and Chen (2001) was adopted. The steps occurred in three stages: design stage, generating the data stage, and interpreting results stage. The design stage was consistent with previous research (Feltz, Chase, Moritz, & Sullivan, 1999). For each run, 10,000 datasets were generated in Mplus 5.2 (Muthén & Muthén, 1998-2007).
Analysis/Results A relatively small sample (N = 200) provides ample π ( > .80) to reject each false null hypothesis. Problematic bias values and coverage values, however, were observed. The relatively small degree of model misspecification appeared to be responsible for the problematic bias values and for the low coverage values.
Conclusions The two questions that commonly arise in validity studies, ″what N do I need to achieve a particular level of π?″ and ″how much π will I have with a fixed N?″, can be empirically determined with Monte Carlo methods.