FIM9 2013 LANT/EPAC

FIM9 Track Performance during 2013 LANT/EPAC seasons

Mike Fiorino, NOAA ESRL, Boulder CO 

Craig Mattocks, the new TECHDEV lead at NHC (I was in this position from 2006-2009) made an interesting comment in an email of 25 Oct 2013:

... "I have heard good things from the Hurricane Specialists this year about the FIM(9), especially the track accuracy."

To see if the 'good things' can be seen in the standard metric of track skill, I looked at the  errors in the atLANTic (LANT) and Eastern north PACific (EPAC) using the working best tracks.

2013 atLANTic mean forecast ('track') error for HWRF, GFS, FIM9

For forecast times (as they say in .mil 'taus') 0-72 h, FIM9 did have slightly higher errors than the GFS.  Note the considerably poorer performance of HWRF which is telling because all three models used the exact same initial (and lateral boundary for HWRF) conditions.  The errors at tau 120h only represent the performance for one storm (09L - HU Humberto). 

At tau 72 h there are more storms:

2013 LANT tau 72 storm-by-storm mean forecast error [nmi]

Most of the GFS improvement over the FIM9 at tau 72 h comes from storm 11L - TS Jerry (7 cases), whereas the FIM9 was better for storms 04L, 05L and 10L with 1,3,4 cases respectively.  

The statistics don't show a clear advantage of FIM9 over the GFS in the LANT, yet the hurricane specialists were impressed(?).  Do we need a better way to measure 'good things.' 

The results in the EPAC are different:

2013 EPAC mean forecast error [nmi] for HWRF, GFS, FIM9

Here we find the FIM9 has lower errors than both HWRF and GFS at all taus.  Also note how the GFS errors grow faster than in FIM9.  One speculation is that while both GFS and FIM9 share a common set of physics, their dynamical cores are substantially different. The precipitation fields in my TCgen site http://ruc.noaa.gov/hfip/tcgen look different with GFS generally having more intense and smaller-scale precipitation events.

It's also impressive that HWRF has lower errors at the longer taus (72-120h) compared to its host global model the GFS.  This error growth is counter intuitive in that the lateral boundaries in HWRF should cause greater errors.  Further investigation is needed, but given the better errors for FIM9 maybe it is in EPAC that the specialists are impressed...

For completeness here are the mean tau 72 errors by storm:

2013 EPAC storm-by-storm tau 72 mean forecast errors [nmi]

Storms 03E (5 cases) and 05E (3 cases) are noteworthy in that FIM9 has lowest errors for 03E but higher errors for 05E

Let's combine the basins for stats in EPAC & LANT:

EPAC & LANT 2013 mean forecast errors [nmi] HWRF v GFS v FIM9

No comparison is complete without including the 'gold standard' of TC track prediction - ECMWF HRES (this is how ECMWF refers to their hi-resolution deterministic run).  There were 8 cases of serious errors in the ECMWF trackers - failure to track the observed storm initially and making large track changes in the first 24 h (with implied speed of motion > 100 kt).  The main problem storm was 02L in which the tracker jumped into the EPAC from the Bay of Campeche.  These errors will be the subject of an upcoming post...but my verification code had to be improved to toss out these cases.  Fortunately no cases > 48 h were removed and most of the bad tracks were for tau 0-24.

For EPAC/LANT combined:

EPAC & LANT 2013 mean forecast error HWRF v GFS v FIM v ECMWF

 ECMWF is clearly the gold standard at tau 48-120 h

The results in the LANT are in some ways more dramatic:

LANT 2013 mean forecast error [nmi] HWRF v GFS v FIM9 v ECMWF

The large initial position errors and the relative poorer performance for tau 0-24 are explained by deficiencies in the ECMWF tracker and no special treatment of TCs in their data assimilation system.  However, the stats at tau 48 and 72 are hugely better than the US models.  A mean forecast error of 90 nmi may be a record low.  When I started in this business in the late 1970s the typical 72-h mean error was around 350 nmi!!!!

In EPAC we find a similar pattern - ECMWF better at the medium and long-range taus (72-120 h)

EPAC 2013 mean forecast error [nmi] HWRF v GFS v FIM9 v ECMWF