In many climatic regimes interannual variability plays an important role in both natural ecosystems and human societies. The extent to which climate models realistically capture observed variability is therefore important. To gauge how model performance has improved over time, we evaluate extratropical modes of variability in the three most recent phases of the Coupled Model Intercomparison Project (CMIP3, 5, and 6). We employ a suite of high-level metrics that objectively gauge the consistency between models and available observations to evaluate how well climate models simulate the observed Northern Annular Mode (NAM), the North Atlantic Oscillation (NAO), the Pacific North America pattern (PNA), the Southern Annular Mode (SAM), the Pacific Decadal Oscillation (PDO), the North Pacific Oscillation (NPO), and the North Pacific Gyre Oscillation (NPGO). We compare results from several commonly applied methodologies and highlight their strengths and weaknesses. As a preliminary result, we show that the extratropical modes are better represented in the newer climate models from CMIP6, especially for their spatial patterns with less improvement in amplitude. In addition the metrics are applied to all available simulations from coupled and uncoupled (i.e., Historical and AMIP, respectively) experiments. We find that in many situations AMIP simulations appear to be suitable for diagnosing errors in atmospheric extratropical modes.