CMC > NCEP 20200130 Update

CMC pulls ahead of the GFS? We're #4?

Update on 20200130: CMC better in Summer Hemisphere?

Mike Fiorino

30 January 2020

Recap of  the November Blog

In my first post:  https://wxmapstertc.blogspot.com/2019/11/cmc-ncep.html I discussed how the CMC GDPS (Global Deterministic Prediction System -- the Canadian model) was outperforming the NCEP GFS (the American model) in a very basic measure of global model forecast quality -- the 5-day northern Hemisphere (NHEM) height Anomaly Correlation (5DNACC). 

I acknowledged that a few points do not make a trend, but at the time hypothesized that the improvements in Canadian model came from new/better physics ...

I asked two questions:

1. when/what were the apparent changes in the CMC GDPS (Global Deterministic Prediction System)?
2. was the gap between the NCEP.GFS and the CMC.GDPS a blip or a trend?

Ron McTaggart-Cowen (ron.mctaggart-cowan@ec.gc.ca) answered #1: 3 July 2019.

And he provided these links describing the changes:

• science: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019MS001781
• tech: http://collaboration.cmc.ec.gc.ca/cmc/cmoi/product_guide/docs/changes_e.html

The most significant physics update to me involved the convective parameterization (adding momentum transport).  As a long-time tropical cyclone (TC) NWP modeler, the impact of better convection was not surprising and consistent with related changes at ECMWF in 2008 that lead to a dramatic improvement in their TC track forecasts (see:https://www.ecmwf.int/sites/default/files/elibrary/2009/17493-record-setting-performance-ecmwf-ifs-medium-range-tropical-cyclone-track-prediction.pdf)

 

Recent Trends...

Pete Kaplan's EMC Stat Page (the long-running web page used at all global model meetings at EMC during my time there in the 1990s) has been down for 'technical reasons'(?), but here are the latest stats from the EMC VSDB (https://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/):

CMC is ahead of the GFS by 0.4 points (a point in NWP is a percentage point) in the NHEM but 1.2 in the SHEM.  A 1.0 point change is a pretty big deal, but again is not necessarily a trend.  What's more impressive  is how the delta is consistent and how the CMC.GDPS does not 'dropout' as badly as the GFS, e.g., the SHEM 11 JAN and NHEM 4 JAN dropouts.

The new CMC GDPS model has been running since July 2019 so we have 7 months to compare in the text table below:

               NHEM                   SHEM
         CMC    GFS  CMC-GFS    CMC    GFS  CMC-GFS
201906: .866   .876  -0.010     N/A  
201907: .863   .871  -0.008    .883   .889  -0.006
201908: .897   .890  +0.007    .882   .891  -0.009
201909: .876   .874  +0.002    .885   .891  -0.006 
201910: .898   .894  +0.004    .908   .899  +0.009
201911: .907   .911  -0.003    .914   .911  +0.003    
201912: .904   .911  -0.007    .903   .903  +0.000
202001: .913   .907  +0.004    .887   .875  +0.012 

What's most interesting is how the Canadian model does better in the summer hemisphere. 

We're not quite at the peak of austral TC activity (around February), but it looks like the convection changes are really improving the CMC height scores and confirms (IMHO) how the tropics impacts the quality of the midlatitude forecasts.

 

Some final thoughts...

Cliff Mass' recent blog post on US NWP (https://cliffmass.blogspot.com/2020/01/the-future-of-us-weather-prediction.html) makes a compelling argument that American operational NWP is fundamently broken with perhaps the greatest dysfunction (unstated there) in the US Navy 😟.  In the NHEM plot above, the Navy global model is barely competitive with the CFSR!  This is very distressing to me as a retired Naval Oceanography officer that implemented the first operational two-way interactive, moving nested-grid TC model in 1981...

I'm seeing a real trend here and at the rate we're going, American NWP will be 4th rate for the foreseeable...

While Cliff correctly identifies our problem mostly as one of leadership (I agree), the (much?) bigger in my mind is the data handling systems at NCEP NCO.  Using the unix filesystem to 'manage' gridded fields is appalling primitive and at least 30 years behind the rest of the world!  Although the Navy global model is clearly in last place, the data systems at FNMOC have always been top notch. 

Data handling is certainly not a sufficient condition, as demonstrated by the lack of performance by the Navy global model, but it is a necessary condition.  I cannot see American NWP at NCEP progressing without a fundamental change in the nitty gritty of data, a fully-funded and outside-DC EPIC notwithstanding.

The Usual Caveats

These comments/opinions are wholly mine and do not refect those of my current employer The University of Colorado Boulder or my previous employer NOAA.

CMC > NCEP?

CMC pulls ahead of the GFS? We're #4?

CMC Anomaly Correlation solidly #3 since August 2019...Waz Up Canada? Eh?

Mike Fiorino

20191125

The NWP league table

I casually follow the NWP scores at: EMC Stat Page (Pete Kaplan's long-running web page used at all global model meetings at EMC during my time there in the 1990s).

Here are the latest 5-d NHEM stats:

The  usual pecking order, or what I consider the 'league table' is:

1. ECMWF
2. UKMO
3. NCEP (GFS)
4. CMC
5. NAVY 

i.e., we (the American Model) are typically #3. Since August CMC has pulled ahead of the GFS and as of today is 1 point higher.  My big question is how the CMC global model/data assimilation system has changed...

Taking a longer term view:

The color scheme: ECMWF; UKMO; CMC; Navy

ECMWF has been ichi ban for over 20 y...the MetOffice #2 and CMC almost always #4.

 

Is CMC the new #3?  How good is the ACC?

I appreciate that the anomaly correlation (ACC) does not measure the entire quality of an NWP modeling system...  However, I hold that the 5-d 500 mb NHEM ACC (5NACC) is a kind of 'magic number' -- highly correlated with skill in other areas/forecast times/variables as seen in the 'scorecard.'

For example, from the latest ECMWF implementation in June 2019:

https://www.ecmwf.int/sites/default/files/elibrary/2019/19156-newsletter-no-160-summer-2019.pdf

Again, while this single score only tells part of the story, general model skill does follow the 5NACC -- it's a necessary but not sufficient condition for model improvement.

The importance of this score in NWP was made very clear to me during my 1.5 year secondment to ECMWF 1998-99 to work on the ERA-40 reanalysis.  I developed a scheme to assimilate the tropical cyclone (TC) 'vitals' (working best track data on TC position, movement, intensity and other structure parameters).  I tested the scheme in the full ERA-40 version of the IFS and a 1point degradation in the 5NACC was the reason why TCs were not assimilated.  At the time, there was zero tolerance for any model change lowering the 5NACC (still true today?)...

 

Are TC forecasts consistent with the 5NACC?

I was curious if the apparent improvement implied by the 5NACC was reflected in TC track prediction...

The short answer: maybe in the atLANTic but not in Western north PACific.

Historically the CMC global model has not been a very good TC forecast aid and is known to have hyper-active tropical convection that causes excessive TC genesis especially in WPAC.  The same is true in EPAC.  The CMC model has shown some track prediction skill in the LANT relative to the GFS.

My standard score for TCs is the 72-h (3-d) mean position error (3MPE)because roughly 2/3 of all official forecasts will have a verifying 72-h position and beacause 3-d is about 1/2 of the mean life cycle of a TC.

In 2018 the LANT 3MPE:

• CMC: 122 nmi (85 cases)
• GFS: 105 nmi (85 cases) --  GFS lower (better) by 17 nmi

For the period 2019081500-2019112500 (today)

• CMC: 128 nmi (73 cases)
• GFS: 143 nmi (73 cases)  -- GFS higher (worse) by 15 nmi

 Here are the full position error plots (i.e., where the numbers above come from) :

2018 CMC v GFS LANT

2019081500-2019112500 CMC v GFS LANT

In WPAC the 3MPE for the same periods as in the LANT

In 2018 WPAC 3MPE:

• CMC: 142 nmi (179 cases)
• GFS: 119 nmi (179 cases) -- GFS lower (better) by 23 nmi

For the period 2019081500-2019112500 (today) in WPAC:

• CMC: 171 nmi (89 cases)
• GFS: 130 nmi (89 cases)  -- GFS lower (better) by 41 nmi

Some Bottom Lines:

I also read the area forecast discussion put out by WFO Denver/Boulder and have recently found references to the Canadian model as part of their prognostic reasoning.  Are the forecasters seeing the improved CMC model?

Why is the new GFS struggling to keep up with the NWP leaders in the UK; and now our friends to the North?

I could offer a few reasons, but to me the big one is that modern (> 2000) NWP is almost entirely a problem of physics, not dynamics (spatial resolution at the hydostatic limit is only significant in its interaction with physics).  Furthermore, observations and data assimilation only matter when the innovations are small (i.e., a good model that 'looks' like the obs).

At the end of the day it's (still) the model and that means physics.