WPAC 2013 -- GFS-based Models v ECMWF

 TC Performance of GFS-based US Models v ECMWF 

2013 WPAC season

Mike Fiorino, NOAA ESRL, Boulder CO 
05 November 2013

Now the that 2013 in the western North Pacific (WPAC) basin is nearly over, I've prepared preliminary error statistics for the tropical cyclone (TC) forecasts of three GFS-based US models v the ECMWF HRES (the 10-d, High-RESolution deterministic run of the IFS).  

These results are preliminary for two reasons:

1. as of time of this writing two systems are active:  30W and 31W (haiyan).  31W is the 7th typhoon to undergo 'ED' - explosive deepening which is defined as a 50 kt change in 24 h.  When you add in RI - rapid intensification (30 kt / 24 h) storms, 11 of 31 typhoons have made large intensity changes.  Further,  with the recent promotion of 31W, there have been 5 'supertyphoon' -- max winds >= 130 kt
2. I use the 'working' best track for verification vice the 'final' best track.  The final, post-season reanalysis of the fix data that makes the true or final best track is typically not available until around Feb/Mar for NHC whereas JTWC finalizes the best track during the season.

The models considered are (ATCF 4-char name in caps):

• HWRF - a three-grid, high-resolution limited-area model that use the GFS for initial and lateral boundary conditions. the grids have a horizontal resolution (dx) of 27:9:3 km
• AVNO - the GFS global model dx~21 km (a spectral model formerly know as the 'aviation' run)
• FIM9 - the ESRL global model dx=15 km (finite-volume, flow-following icosahedral dynamical core)
• EDET - ECMWF HRES, dx~15 km (a spectral model with 129 layers)

Table 1. test of tables from google docs

a	b
e	d

from google doc + snagit grab

First consider the mean forecast (track) error, defined as the great-circle distance between the best track and forecast position, but before looking at the stats, the verification system needs to be reviewed.   

There were cases of tracker failures (inability to accurately locate the model TC) that had to be eliminated to avoid skewing the statistics.  These failures are obvious when plotting the tracks, as is done in operations, and are few in number in WPAC.  Details are given in appendix A.

Figure 1.  WPAC 2013 mean forecast error [nmi]

There are no clear winners in that no single model has lower numbers at all forecast times or 'taus'.  For the longer taus 96 and 120 h (day4 D4 & D5) there is greater separation because of fewer storms going into the mean --  one bad storm can ruin the mean.  We'll dig into the details later...

A second measure of forecast performance is storm 'intensity' or a comparison of the model max surface wind speed (Vmax) to 'observed' from the (working) best track.  The standard metric for intensity is the mean absolute error (no directionality), but from a modeling perspective the mean error itself, i.e., the bias, may be more important.  Although bias is not commonly displayed, I do below:

Figure 2.  WPAC 2013 intensity or Vmax eror: lines are the mean absolute error and the bars the mean error or bias

The bars show bias and the lines the mean absolute error.  In some ways the numbers are typical but other not:

• HWRF has the highest spatial resolution (3km) and takes great care in initializing the model vortex.  Consequently, the model has near zero initial bias and shows only a slight over intensified bias out to 120 h (5 kt). More remarkable is the mean abs error of 15 kt at 72 h.  One of the best statistical-dynamical intensity aids is LGEM (Logistic Growth Equation Model) and this aid serves as a baseline for performance.  In a head-to-head comparison at 72 h: HWRF: 16|1 kt v LGEM: 19|-12 kt [MM|BB where MM is the mean abs error and BB is the bias].  A 15 kt mean abs error is simply excellent and these statistics are among the best I have ever seen for any model, numerical or statistical, in WPAC.
• The initial ECMWF mean|bias is 20|-18 kt.  These statistics imply a very poor analysis of the cyclone intensity.  The bias does decrease in time and is only -3 kt at tau 120 h.  This drop in bias is typical for the ECMWF model and in the past there were cases where the bias goes positive at 120 h.  The basic conclusion is that the ECMWF data assimilation system does a (very) poor job in analyzing the TC vortex.
•  The two GFS-based global models: AVNO and FIM9 both under analyze and under predict TC intensity.  The mean abs error is about 5 kt higher than HWRF at all taus and most of the error comes from a negative (weak-storm) bias.
• Large intensity errors do not imply large track errors.  The key result of my PhD work at the Naval Postgraduate School (Fiorino and Elsberry 1989) was that the TC inner-core, where the intensity change occurs, does not affect the dynamics of vortex motion, i.e., the scales that dominate the motion process are much larger the the TC inner core (~50 km).  Nonetheless, the large intensity error in the ECMWF model may reflect vortex structure errors on larger scales.  We will explore this relationship in a separate blog...
• FIM9 with higher horizontal resolution than the GFS (AVNO) has less bias than AVNO despite the tracker using the same grid (0.5 deg global).  The higher model resolution does permit more intense cyclones and if we tracked at the native resolution, the bias would probably be less...

One way to compensate for large intensity errors in the numerical models when forecasting, is to apply statistical post-processing.  The scheme used at both JTWC and NHC is to calculate an initial 'offset' and then add a portion of the inital offset to the forecast.  The offset is defined as the intensity analyzed in operations minus the model initial intensity.  This operational intensity is generally not the same as the working best track, but since all intensity is analyzed to the nearest 5 kt, does not make much of a practical difference, but does result a some initial intensity error.  For global models, I apply 100% offset at tau 0 and 0% at tau 72 h, with a linear variation between 0-72 h.  The limited-area models set the 0% at tau 24 h.

Here are the bias-corrected statistics:

Figure 3. WPAC 2013 intensity or Vmax eror with bias correction. lines are the mean absolute error and the bars the mean error or bias

The big initial under bias for ECMWF has been eliminated and the correction has greatly reduced the size of the bias from tau 0-48 h and the mean abs errors. Still, the global model errors are higher the HWRF (and LGEM, not shown).

Appendix A - details of the stats

  
Figure 1 plots these numbers

                      000    012    024    036    048    072    096    120  
           HWRF      10.1   31.0   43.7   58.0   74.9  106.2  165.4  276.9
           AVNO      13.5   27.0   39.5   54.8   76.2  113.5  172.6  237.5
           FIM9      14.7   25.5   40.6   56.8   74.9  110.9  165.1  212.3
           EDET      22.0   27.2   39.2   55.5   68.4  108.5  178.2  247.3
           CONW       7.5   28.3   45.0   63.5   78.8  118.6  155.9  220.2
         #CASES       206    189    172    151    132    95     60     36    
 #Tossed( HWRF)       2      1      1      1     
 #Tossed( AVNO)       1      1      1      1     
 #Tossed( FIM9)       1      1     
 #Tossed( EDET)       1      2      2     

#Tossed is the number of cases removed at that forecast tau

model runs that were filtered:

BE filter Cases for: HWRF
stmid     dtg         tau     BE[nmi]
02W.2013  2013022018   12     221
14W.2013  2013083100   12     253

BE filter Cases for: AVNO
stmid     dtg         tau     BE[nmi]
14W.2013  2013082912   12     203

BE filter Cases for: FIM9
stmid     dtg         tau     BE[nmi]
02W.2013  2013022100    0     173
02W.2013  2013022100   12     335

BE filter Cases for: EDET
stmid     dtg         tau     BE[nmi]
08W.2013  2013071700    0      190
08W.2013  2013071612   12     217

adsfasdf