Evaluation of the performance of CMIP5 and CMIP6 models in simulating the South Pacific Quadrupole—ENSO relationship

Zhenchao Wang , Lin Han , Ruiqiang Ding , Jianping Li

a School of Atmospheric Sciences, Chengdu University of Information Technology, Chengdu, China

b State Key Laboratory of Earth Surface Processes and Resource Ecology, Beijing Normal University, Beijing, China

c Frontiers Science Center for Deep Ocean Multispheres and Earth System (FDOMES), Key Laboratory of Physical Oceanography, Institute for Advanced Ocean Studies,Ocean University of China, Qingdao, China

d Laboratory for Ocean Dynamics and Climate, Pilot National Laboratory for Marine Science and Technology, Qingdao, China

Keywords:South Pacific quadrupole ENSO CMIP5 and CMIP6 models Extratropical atmospheric forcing

A B S T R A C T The South Pacific Quadrupole (SPQ) is the extratropical South Pacific’s second principal sea surface temperature mode. Previous observational studies have shown that the SPQ promotes the onset of the El Ni?o—Southern Oscillation (ENSO). The present study evaluates and compares simulations of the SPQ—ENSO relationship by 20 climate models from CMIP6 and their corresponding 20 previous models from CMIP5. It is found that 16 of the 20 pairs of models are able to consistently reproduce the spatial pattern of the SPQ. In terms of simulating the SPQ—ENSO relationship, 9 of the 16 CMIP6 models show significant improvement over their previous CMIP5 models. The multi-model ensemble (MME) of these 16 CMIP6 models simulates the SPQ—ENSO connection more realistically than the CMIP5 MME. Further analysis shows that the performance of the model simulations in reproducing the SPQ—ENSO relationship is strongly dependent on their ability to simulate the SPQ-related surface air—sea coupling processes over the southwestern and southeastern South Pacific, as well as the response of the SPQ-related equatorial subsurface ocean temperature anomalies. The improvement of the CMIP6 models in simulating these two processes is responsible for the improved performance of the CMIP6 models over their CMIP5 counterparts in simulating the SPQ—ENSO relationship.

1. Introduction

The South Pacific Quadrupole (SPQ) mode is the extratropical South Pacific’s second principal mode of sea surface temperature (SST) variability, which is the response of the ocean to the forcing of the first Pacific—South American ( Mo, 2000 ) —like atmospheric variability( Ding et al., 2014 ). Variability of the SPQ has been shown to cause climate anomalies in many parts of the globe. For example, it can affect precipitation in the Pacific Intertropical Convergence Zone ( Qin et al.,2018 ; Li et al., 2020 ) and modulate the interdecadal variability of ozone over the Southwest Pacific ( Zhang et al., 2019 ).

In particular, the SPQ promotes the occurrence of El Ni?o—Southern Oscillation (ENSO) events ( Ding et al., 2014 ). Specifically, the positive SPQ mode reaches its maximum intensity during the austral early autumn (February—March—April; FMA), when the associated SST anomalies (SSTAs) manifest as a zonal quadrupole SSTA pattern in the South Pacific, with centers over the extratropical southwestern South Pacific(SWP), the Ross Sea, the Bellingshausen Sea, and the extratropical southeastern South Pacific (SEP) (Fig. S1(a)). Subsequently, the cold SSTAs in the SWP extend northwestward through the positive wind—evaporation—SST (WES) feedback ( Xie and Philander, 1994 ) and cause an anomalous westerly over the equatorial western Pacific (Fig. S1(c)). In addition, the warm SSTA pattern associated with the SPQ over the SEP is similar to the South Pacific meridional mode (SPMM; Zhang et al., 2014 ) during FMA. This SPMM-like warm SSTA pattern is then also extended to the east-central equatorial Pacific by the positive WES feedback, enhancing the warming in the east-central equatorial Pacific. This increases the zonal SST gradient in the west-central equatorial Pacific, (also across the east-central equatorial Pacific), which consequently intensifies the anomalous westerlies over the equatorial western Pacific and forces the anomalous easterlies in the equatorial eastern Pacific (Fig. S1(d, e)),which consequently intensifies the anomalous westerly over the equatorial western Pacific (Fig. S1(d, e)), triggering positive Bjerknes feedback( Bjerknes, 1969 ) in the equatorial Pacific. Furthermore, the anomalous westerly winds in the western Pacific (anomalous easterly winds in the eastern Pacific) caused by the SPQ may excite eastward (westward)-propagating downwelling equatorial Kelvin (Rossby) waves, deepening the thermocline of the east-central tropical Pacific and thus increasing warming there (Fig. S1(j)). During SON (0) (September—October—November of the given year), the warming in the east-eastern Pacific is further enhanced by the arrival of equatorial Kelvin waves from the west and Rossby waves from the east (Fig. S1(k)). Eventually, an El Ni?o event develops in the equatorial Pacific, approximately three seasons after the SPQ peaks (Fig. S1(f)).

The previous studies mentioned above demonstrate that the SPQ is a basin-scale SST mode that can effectively act as a tracer for ENSO events.Recently, Shi and Ding (2020) showed that incorporating the effects of the SPQ can improve the prediction of ENSO. However, it is unclear how well current climate models perform in simulating the link between the SPQ and ENSO. Therefore, the purpose of the present study is to address(1) whether the models of phase 5 of the Coupled Model Intercomparison Project (CMIP5) and their successors in phase 6 (CMIP6) can simulate the observed significant SPQ—ENSO relationship, and whether the CMIP6 models display improved performance in simulating the SPQ—ENSO relationship compared to their CMIP5 predecessors; and (2) the critical contributors to model performance in simulating the SPQ—ENSO connection.

The remainder of the paper is organized as follows: Section 2 introduces the data and methods. Section 3 evaluates the capability of the CMIP5 and CMIP6 models to reproduce the SPQ—ENSO relationship.Section 4 discusses the factors affecting the models’ simulations of the SPQ—ENSO relationship. And finally, Section 5 provides a summary.

2. Data and methods

2.1. Data

The monthly SST data in this study are from the Hadley Centre Sea Ice and SST dataset ( Rayner et al., 2003 ). The surface winds are from the National Centers for Environmental Prediction—National Center for Atmospheric Research Reanalysis 1 dataset ( Kalnay et al., 1996 ) for the period 1950—2017. The subsurface ocean temperature data are from the Simple Ocean Data Assimilation, version 2.2.4 ( Carton and Giese, 2008 )for the period 1950—2010. The monthly outputs of 20 CMIP5 models and their CMIP6 successors were downloaded from their first run of the pre-industrial simulation experiments ( Taylor et al., 2012 ; Eyring et al.,2016 ). The key information regarding the models is listed in Table S1. Since some models do not produce 10-m surface wind, we follow You and Furtado (2018) and adopt 1000-hPa wind in the models as a proxy for 10-m surface wind. In addition, since the simulation time lengths vary among models, in this study we take the last 100 years of each model’s simulation outputs for analysis. In addition, for analysis based on the multi-model ensemble (MME) mean, all simulations are bilinearly interpolated to a 1.0°×1.0°longitude—latitude grid.

2.2. Methods

Statistical significance for the regression and pattern correlations is tested with the two-tailed Student’s

t

-test, where the number of effective degrees of freedom

N

is calculated according to Bretherton et al. (1999) :

where

N

represents the effective sample size, and

r

and

r

represent autocorrelation coefficients with a lag of one for two variables

x

and

y

,respectively.

2.3. Index definitions

According to Ding et al. (2014) , the SPQ is defined by the second mode of the empirical orthogonal function (EOF2) of the monthly SSTAs in the South Pacific (15°—60°S, 140°E—70°W). The SPQ index(SPQI) is defined by the normalized principal component (PC2) linked to EOF2. Since the SPQ tends to reach its maximum strength in FMA( Ding et al., 2014 ), we apply the FMA-averaged SQPI to represent the SPQ variability. Following You and Furtado (2018) , we apply maximum covariance analysis ( Bretherton et al., 1992 ) to the SST and 10-m wind anomalies over the region (35°—10°S, 180°—70°W) to obtain the SPMM_SST index, which is used to represent the variability of the SPMM. The SWP SST pattern (SWP_SST) index is defined by the SSTA averaged over the region (47°—25°S, 142°E—179°W) ( Ding et al., 2014 ).The warm water volume (WWV) is a convenient index to represent the heat content of the equatorial Pacific, and the eastern tropical Pacific WWV (EWWV) is defined by the volume of water above the 20 °C isotherm from 5°S—5°N and 155°—80°W ( Meinen and McPhaden, 2000 ).The variability of ENSO is expressed by the Ni?o-3.4 index, which is calculated by the averaged SSTA from 5°S—5°N and 170°—120°W( Anderson, 2007 ; Deser et al., 2012 ). Additionally, to eliminate possible effects of the ENSO cycle on the SPQ—ENSO relationship, linear regression is used to remove the variability of all monthly average variables that are linearly correlated with the FMA-averaged Ni?o-3.4 index for the same period. With respect to a given year (0), the following year is termed (1).

3. Performance of CMIP5 and CMIP6 models in simulating the SPQ-ENSO relationship

3.1. The SPQ reproduced in CMIP5 and CMIP6 models

To assess the relationship between the SPQ and ENSO in the CMIP5 and CMIP6 models, we first evaluate the ability of the models to simulate the SPQ. Only models reproducing the SPQ are used to further examine their performance in simulating the SPQ—ENSO relationship in the next section.

We perform an EOF analysis of the SSTAs in the South Pacific from observations and the 20 CMIP5 models and their CMIP6 successors, and the results are displayed in Figs. S2 and S3. It is found that the SPQ in the models can be characterized with the EOF2 or EOF3 pattern. The EOF1 pattern exhibits a tripolar-like SSTA pattern (not shown), which is closely related to ENSO ( Kwok and Comiso, 2002 ; Wang et al., 2007 ;Terray, 2011 ); plus, it also effectively regulates precipitation in eastern China during the boreal spring and autumn ( Li et al., 2014 , 2018 ). We then examine the spatial correlation coefficients between that observed for EOF2 and that simulated for EOF2 and EOF3 ( Fig. 1 ). Some models are found to have simulated EOF2 patterns that are significantly correlated to the observed EOF2 pattern, while their simulated EOF3 patterns are not, in which case we define the simulated EOF2 pattern as the SPQ mode and its corresponding normalized PC2 as the SPQI. Likewise,some models simulate EOF3 patterns that are significantly correlated to the observed EOF2 pattern, while the simulated EOF2 patterns are not,in which case we define the simulated EOF3 pattern as the SPQ mode and its corresponding normalized PC3 as the SPQI. Other models simulate both EOF2 and EOF3 patterns that are significantly correlated with the observed EOF2 pattern. In this case, we define the simulated EOF2(EOF3) pattern as the SPQ, when the correlation coefficient of the simulated EOF2 (EOF3) with the observed EOF2 is higher than the correlation coefficient of the simulated EOF3 (EOF2) with the observed EOF2.There are also some models when neither EOF2 nor EOF3 simulated patterns that are significantly correlated with the observed EOF2 pattern,which means these models cannot reproduce the SPQ. In summary, the EOF2/EOF3 patterns from 17 CMIP5 models and 19 CMIP6 models are similar to the observed EOF2 pattern, suggesting that these models successfully represent the SPQ with the quadrupole pattern of SST variability. However, the other four models (CMIP5: CESM1-WACCM, FGOALSs2, INMCM4; CMIP6: NorESM2-LM) fail to simulate the SPQ, so these four models are not further analyzed. In addition, in order to be able to compare the CMIP5 models and their CMIP6 successors one by one in terms of their ability to simulate the SPQ—ENSO relationship, these four models (CMIP5: NorESM1-M; CMIP6: CESM2-WACCM, FGOALS-f3-L, INM-CM5-0) are also not further analyzed in the next section ( Fig. 1 ;shaded models). This means that, ultimately, we select 16 CMIP5 models and their CMIP6 successors for subsequent analysis (the specific EOF modes chosen to represent the SPQ for each model are listed in Table S1). The observed SPQ pattern and the SPQ pattern simulated by our selected 16 CMIP5 models and their CMIP6 successors, are shown in Fig. S4.

Fig. 1. Spatial correlation coefficients between the observed second leading EOF (EOF2) and simulated EOF2 (blue bars) and EOF3 (red bars) from each of the (a)16 CMIP5 models and (b) 16 CMIP6 successor models. The black asterisks indicate statistical significance at the 95% confidence level.

3.2. Simulated SPQ-ENSO relationship

In this section, the performance of the selected CMIP5 and CMIP6 models in simulating the SPQ—ENSO relationship is evaluated. Fig. 2 shows the following austral summer (ND(0)J(1)) SST and surface wind anomaly field from the observation and the 16 CMIP5 and CMIP6 models regressed onto the normalized FMA(0)-averaged SPQI. To further quantify the simulated SPQ—ENSO relationship, we calculate the regression coefficients for the FMA(0)-averaged SPQI and the ND(0)J(1)-averaged Ni?o-3.4 index for each model and the MME mean, as shown in Fig. 3 .

Fig. 2. Regression maps of the FMA(0)-averaged SPQI with the following ND(0)J(1)-averaged SST (units: °C; shaded), and surface wind anomalies (units: m s —1 ;vectors) in the (a) observation and (b1—b16) 16 CMIP5 models, (c1—c16) 16 CMIP6 models, and the MME mean of (b0) CMIP5 and (c0) CMIP6. Only SST and surface wind anomalies significant at the 95% confidence level are shown.

Fig. 3. (a) Regression coefficients of the FMA(0)-averaged SPQI with the ND(0)J(1)-averaged Ni?o-3.4 index in the 16 CMIP5 models (blue bars), 16 CMIP6 models(red bars), and the MME means of CMIP5 and CMIP6. The horizontal dashed line shows the 95% confidence level. The error bars in the MME means are calculated as 2 standard deviations (a 95% confidence interval based on the normal distribution) of the 10 000 inter-realizations of a bootstrap method. (b) Histograms of 10 000 realizations of a bootstrap method for the Ni?o-3.4 region SST response. The Ni?o-3.4 region SST response is measured by the regression coefficients of the FMA(0)-averaged SPQI with the ND(0)J(1)-averaged Ni?o-3.4 index. The blue and red vertical lines indicate the mean values of 10 000 inter-realizations for the CMIP5 and CMIP6 models, respectively. The gray shaded regions represent the 1% and 99% percentile of the 10 000 inter-realization.

From the observations ( Fig. 2 (a)) it can be seen that, after the occurrence of the positive phase of the SPQ in FMA, SST and wind anomaly patterns similar to El Ni?o events tend to occur in the equatorial Pacific during the following austral summer. Combining the results of Figs. 2 and 3 , nine of the 16 CMIP6 models ((a1)ACCESSESM1-5, (a4)CESM2, (a5)CanESM5, (a7)FIO-ESM-2—0, (a8)GISS-E2-1-H, (a9)GISS-E2—1-G-CC, (a10)GISS-E2-1-G, (a11)HadGEM3-GC31-MM,and (a13)MIROC6) examined are able to some extent to reproduce results similar to the above observations, and the corresponding regression coefficients are statistically significant at the 95% confidence level.Moreover, these nine models display improved ability in simulating the SPQ—ENSO relationship compared to their corresponding CMIP5 predecessors. Six CMIP6 models and their corresponding CMIP5 predecessors fail to simulate the SPQ—ENSO relationship ((c2)BCC-CSM2-MR-(b2)BCC-CSM1-1-m, (c3)BCC-ESM1—(b3)BCC-CSM1-1), (c6)EC-Earth3—(b6)EC-EARTH, (c12)IPSL-CM6A-LR—(b12)IPSL-CM5B-LR, (c14)MPIESM1-2-LR—(b14)MPI-ESM-LR, (c15)MPI-ESM-MR—(b15)MPI-ESM1-2-HR). The one remaining CMIP6 model (NorESM2-MM) fails to simulate the SPQ—ENSO relationship, while its corresponding CMIP5 model(NorESM1-ME) reproduces it well. When averaged separately for all models in both model groups, the CMIP6 MME-averaged pattern( Fig. 2 (c0)) is closer to the observed result than the CMIP5 MMEaveraged pattern ( Fig. 2 (b0)). The MME-average regression coefficient increases by approximately 50% from 0.127 °C for the CMIP5 models to 0.255 °C for the CMIP6 models ( Fig. 3 (a)), but the difference between the two averages is not statistically significant above the 95% confidence level, according to the bootstrap method ( Fig. 3 (b)). The above results indicate that roughly half of the CMIP6 models show a significant improvement in their ability to simulate the SPQ—ENSO relationship compared to their corresponding previous CMIP5 models, but one CMIP6 model has worsened. Overall, the CMIP6 multi-model result simulates a better SPQ—ENSO relationship than the CMIP5 one.

Fig. 4. (a) Scatterplot for the CMIP5 (blue) and CMIP6 (red) simulations between the SPQ—ENSO correlation and SPMM—ENSO correlation. (b) Scatterplot for the CMIP5 (blue) and CMIP6 (red) simulations between the SPQ—ENSO correlation and SWP—ENSO correlation. (c) Scatterplot with regression fitting plane for the CMIP5 simulations’ correlation coefficients of FMA(0)-averaged SPQI, SPMM_SST, and SWP_SST indices with the ND(0)J(1)-averaged Ni?o-3.4 index. (d) As in (c) but for CMIP6. (e) Scatterplot for the CMIP5 (blue) simulations between the SPQ—ENSO correlation and SPQ—EWWV correlation. (f) As in (e) but for CMIP6 (red). The box enclosed by the dashed red line in the upper-right corner identifies a region in which the correlation coefficients pass the significance test at the 95% confidence level.

4. Possible factors influencing the simulated SPQ-ENSO relationship

The above analysis shows that the CMIP5 models and their CMIP6 successors differ in their ability to simulate the SPQ—ENSO relationship.In previous studies ( Ding et al., 2014 , 2020 ), it was demonstrated that after the SPQ peaks in FMA, the SSTAs associated with the SPQ exhibit a cold SSTA pattern in the SWP and an SPMM-like SSTA pattern in the SEP, both of which play essential roles in the SPQ-induced ENSO process through surface air—sea coupling processes. Also, the response of the SPQ-related equatorial subsurface ocean temperature anomalies is crucial to the contribution of the SPQ—ENSO relationship. Therefore, we suggest that the ability of the models to simulate the forcing of ENSO by the two SSTA patterns (SWP and SPMM-like) associated with the SPQ and the response of the subsurface ocean temperature anomalies associated with SPQ, may be the main reason for the differences between models in simulating the SPQ—ENSO relationship. Next, we explore these aspects in the models.

The correlation coefficients between the FMA(0)-averaged SPMM_SST index or SWP_SST index and the ND(0)J(1)-averaged Ni?o-3.4 index are first calculated, which represent the relationships between the SPMM-like and the SWP SSTAs patterns and ENSO,respectively. We then associate the SPQ—ENSO correlation with the SPMM—ENSO and SWP—ENSO correlations in the CMIP5 and CMIP6 models, the results of which are presented in Fig. 4 (a, b), respectively.The results show a strong positive linear relationship between the SPQ—ENSO correlation and both the SPMM—ENSO and SWP—ENSO correlation in both the CMIP5 and CMIP6 models (the correlation coefficients are all statistically significant above the 95% confidence level). The models that can simulate the SPQ—ENSO relationship tend to have a strong SPMM—ENSO link and a strong SWP—ENSO link.Most models that fail to simulate the SPQ—ENSO relationship also do not simulate the SPMM—ENSO relationship well, or the SWP—ENSO relationship. Next, we measure the contribution of the joint effect of the SPMM and SWP on ENSO to the SPQ—ENSO relationship in the CMIP5 and CMIP6 models. In the CMIP5 models, the correlation based on the joint action of the SPMM and SWP on ENSO with the SPQ—ENSO relationship is as high as 0.82 ( Fig. 4 (c)), while in the CMIP6 models the correlation also reaches 0.79 ( Fig. 4 (d)). The above analysis shows that if a good reproduction of the SPMM—ENSO connection and the SWP—ENSO connection can be achieved by a model, this will then favor the model in providing a reasonable simulation of the SPQ—ENSO relationship. This partly explains why some CMIP6 models display improved ability in simulating the SPQ—ENSO relationship relative to their corresponding previous CMIP5 models, while one CMIP6 model(NorESM2-MM) shows a poorer performance in this regard compared to its CMIP5 counterpart (NorESM1-ME).

In addition, the response of equatorial subsurface ocean temperature anomalies associated with the SPQ is also an important factor in the SPQ—ENSO relationship. As shown in the observations, in the process of SPQ-induced ENSO, the anomalously warm subsurface ocean temperature induced by the SPQ will gradually propagate eastward with the seasons to the equatorial eastern Pacific, resulting in further warming of the local SST and promotion of the onset of ENSO. We therefore compute the correlation between the FMA(0)-averaged SPQI and ND(0)J(1)-averaged EWWV, and link it with the SPQ—ENSO relationship ( Fig. 4 (e,f)). The results show that the models simulating the SPQ—ENSO relationship have a very close correlation with the SPQ—EWWV relationship.Most CMIP5 and CMIP6 models that can simulate the SPQ—ENSO relationship can also simulate the SPQ—EWWV relationship (except for the CMIP6 models ACCESS-ESM1 and CanESM5). Those CMIP5 and CMIP6 models that fail to simulate the SPQ—EWWV relationship also fail to simulate the SPQ—ENSO relationship. This indicates that the response of the SPQ-related equatorial subsurface ocean temperature anomalies plays an important role in the SPQ—ENSO relationship, which is consistent with the results of the previous analysis of the observations. Moreover,the results also show that the CMIP6 models display improved ability in simulating the SPQ—EWWV relationship compared with their CMIP5 predecessors. This is another reason why some CMIP6 models have improved their ability to simulate the SPQ—ENSO relationship relative to their CMIP5 counterparts.

5. Summary and discussion

In this study we evaluate the ability of 20 CMIP5 models and their CMIP6 successors in simulating the SPQ—ENSO relationship. First, we evaluate their capability to simulate the SPQ, from which we establish that three of the 20 CMIP5 models (CESM1-WACCM, FGOALS-s2,INMCM4) and one of the CMIP6 models (NorESM2-LM) fail to do so.Next, we select those models for which both their CMIP5 and CMIP6 versions can simulate the SPQ, and use them to analyze the SPQ—ENSO relationship. The results show that, among the 16 selected CMIP5 models and their CMIP6 successors, in terms of simulating the SPQ—ENSO relationship, nine CMIP6 models are capable of doing so, which is an improvement compared to their previous CMIP5 models, while six CMIP6 models and their corresponding CMIP5 models both fail in their simulations, with the remaining CMIP6 model failing but its corresponding CMIP5 model succeeding. Overall, the simulation performance of the CMIP6 models for the SPQ—ENSO relationship is improved compared to that of the CMIP5 models.

Finally, we explore the factors that affect the ability to simulate the SPQ—ENSO relationship between the models. The SPQ-related surface air—sea coupling processes in the SWP and SEP, as well as the response of the SPQ-related equatorial subsurface ocean temperature anomalies,are found to contribute significantly to the model-simulated SPQ—ENSO relationship ( Fig. 4 (c—f)). This result also confirms the previous conclusion of Ding et al. (2014) , based on observations, that the SPQ induces ENSO mainly through these two processes. The improvement of the simulation of these two processes in the CMIP6 models is responsible for their improved performance in simulating the SPQ—ENSO relationship compared to their CMIP5 predecessors.

Declaration of Competing Interest

No potential conflict of interest is reported by the authors.

Funding

This research was jointly supported by the National Natural Science Foundation of China [Grant number 41975070 ] and the State Key Laboratory of Tropical Oceanography , South China Sea Institute of Oceanology, Chinese Academy of Sciences [Project number LTO1901 ].

Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi: 10.1016/j.aosl.2021.100057 .