CMC pulls ahead of the GFS? We're #4?
CMC Anomaly Correlation solidly #3 since August 2019...Waz Up Canada? Eh?
Mike Fiorino
20191125
The NWP league table
I casually follow the NWP scores at: EMC Stat Page (Pete Kaplan's long-running web page used at all global model meetings at EMC during my time there in the 1990s).
Here are the latest 5-d NHEM stats:
The usual pecking order, or what I consider the 'league table' is:
- ECMWF
- UKMO
- NCEP (GFS)
- CMC
- NAVY
Taking a longer term view:
 | |
| The color scheme: ECMWF; UKMO; CMC; Navy |
ECMWF has been ichi ban for over 20 y...the MetOffice #2 and CMC almost always #4.
Is CMC the new #3? How good is the ACC?
I appreciate that the anomaly correlation (ACC) does not measure the entire quality of an NWP modeling system... However, I hold that the 5-d 500 mb NHEM ACC (5NACC) is a kind of 'magic number' -- highly correlated with skill in other areas/forecast times/variables as seen in the 'scorecard.'
For example, from the latest ECMWF implementation in June 2019:
 |
| https://www.ecmwf.int/sites/default/files/elibrary/2019/19156-newsletter-no-160-summer-2019.pdf |
Again, while this single score only tells part of the story, general model skill does follow the 5NACC -- it's a necessary but not sufficient condition for model improvement.
The importance of this score in NWP was made very clear to me during my 1.5 year secondment to ECMWF 1998-99 to work on the ERA-40 reanalysis. I developed a scheme to assimilate the tropical cyclone (TC) 'vitals' (working best track data on TC position, movement, intensity and other structure parameters). I tested the scheme in the full ERA-40 version of the IFS and a 1point degradation in the 5NACC was the reason why TCs were not assimilated. At the time, there was zero tolerance for any model change lowering the 5NACC (still true today?)...
Are TC forecasts consistent with the 5NACC?
I was curious if the apparent improvement implied by the 5NACC was reflected in TC track prediction...
The short answer: maybe in the atLANTic but not in Western north PACific.
Historically the CMC global model has not been a very good TC forecast aid and is known to have hyper-active tropical convection that causes excessive TC genesis especially in WPAC. The same is true in EPAC. The CMC model has shown some track prediction skill in the LANT relative to the GFS.
My standard score for TCs is the 72-h (3-d) mean position error (3MPE)because roughly 2/3 of all official forecasts will have a verifying 72-h position and beacause 3-d is about 1/2 of the mean life cycle of a TC.
In 2018 the LANT 3MPE:
- CMC: 122 nmi (85 cases)
- GFS: 105 nmi (85 cases) -- GFS lower (better) by 17 nmi
- CMC: 128 nmi (73 cases)
- GFS: 143 nmi (73 cases) -- GFS higher (worse) by 15 nmi
 | |
| 2018 CMC v GFS LANT |
 | |
| 2019081500-2019112500 CMC v GFS LANT |
In 2018 WPAC 3MPE:
- CMC: 142 nmi (179 cases)
- GFS: 119 nmi (179 cases) -- GFS lower (better) by 23 nmi
- CMC: 171 nmi (89 cases)
- GFS: 130 nmi (89 cases) -- GFS lower (better) by 41 nmi
Some Bottom Lines:
I also read the area forecast discussion put out by WFO Denver/Boulder and have recently found references to the Canadian model as part of their prognostic reasoning. Are the forecasters seeing the improved CMC model?Why is the new GFS struggling to keep up with the NWP leaders in the UK; and now our friends to the North?
I could offer a few reasons, but to me the big one is that modern (> 2000) NWP is almost entirely a problem of physics, not dynamics (spatial resolution at the hydostatic limit is only significant in its interaction with physics). Furthermore, observations and data assimilation only matter when the innovations are small (i.e., a good model that 'looks' like the obs).
At the end of the day it's (still) the model and that means physics.
"Furthermore, observations and data assimilation only matter when the innovations are small (i.e., a good model that 'looks' like the obs)." - could you elaborate on that?
ReplyDeletedata assimilation (DA) is the process of correcting the 'background' given observations.
ReplyDeletethe input to DA are the 'innovations' or the difference between the background and the observations. the output is the 'increment' which is added to the background to make the 'analysis'. the analysis is used to make the model forecast (the initial conditions).
the background in modern NWP is a 4-D model forecast typically over a 12-h windows.
if the difference (innovations) is zero, then the DA output (the increment) would be zero as well or 'analysis' = 'background'
of course the innovations are never zero, but when they are small, the DA method is less important.
because the background IS the model, to make the innovations small means that the model 'looks' or is close to the observations.
beyond making small innovations (better model), the other big DA problem is how to handle larger the average innovations... sometimes the ob is wrong but other times it's the model. regardless, large, but not too big, innovations mean new and significant information is there and a good DA will take advantage of these important innovations to improve the analysis (and the model).
but the it's guaranteed that if the modelers make the model close to the obs, the analysis will improve and so will the forecast.
thanks for your question and best /R Mike
NB: the background was known in the early days as the 'first guess'