prospectus for a WAF paper
- why metrics matter: "you're only as good as what you measure"
- .gov & .mil set standards for operational forecast quality or goodness
- GPRA for NWS - mean 48-h PE/IE
- PACOM for .mil - mean 24,48,72,120 PE
- model 'goodness' based on mean PE/IE statistics
- TC forecast is...
- surface (10 m) wind field
- 2-D functional representation using Position (lat/lon), Intensity (Vmax) and Radii (R34/50/64) POCI/ROCI (pressure and radius of outermost closed isobar) and other parameters...
- NWS (noaa.gov) & PACOM (.mil) warnings/advisories based on onset of 34 kt winds
- Standard TC metrics:
- PE - position error falsely called 'track' error
- IE - intensity error; intensity defined by Vmax not Pmin
- forecast taus 0,12,24,36,48,60,72,96,120
- primary statistic is the mean
- best track uncertainty:
- P ~ 5-20 nmi depending on I (big I small P uncertainty)
- I ~ 10-20 kts largest for small I and (ironically) very large I
- Properties of PE (and IE)
- first and foremost NOT equivalent to NWP metrics like 5-day 500 mb NHEM AnomCorr (NAC)
- time series of 5DNAC --> mean
- continuous from a continuous process (the model)
- # of cases the same for day 5, day 10, day...
- time series of 24/72/120 h PE -- 2019 LANT for hwrf,avno,tecm5
- discontinuous
- 2 or more PE at a given time (2 or more storms)
- # of cases at each tau is different showed by histogram of R34 at tau 24/72/120
- varies with basins
- in the LANT only 1 of every 2 forecasts has a 72-h verifying position
- show how 24-h PE using only 120 h storms != as 24-h PE using all possible
- display means of both NAC and PE as die off curves
- NAC dieoff can be differentiated PE dieoff cannot
- PE should be displayed for each tau separately!
- # of storms / mean PE
- apples v orange problem!!
- the 'population' is season/basin dependent
- year-to-year variability in season/basin mean implies the population cannot be well defined
- serial correlation between forecasts reduces number of cases
- e-folding time ~ 12-18 h or for forecasts every 6-h Nind~Nall/3
- for every forecast...
- number of verifying cases at:
- tau 24 ~ 80% (short range)
- tau 72 ~ 50% (medium range)
- tau 120 ~ 30% (long range)
- mean PE/IE represent a subset of storms
- contribution by storm highly variable
- the most important part of the forecast is track
- 80% v 20%?
- How to improve mean PE?
- separate from IE
- a model must make a 'good' track forecast
- to use the intensity? maybe...but physically intensity does depend on track
- can be seen in ensembles -- need to make this plot again...
- improve the process that generates the forecast -- the model -- why ECMWF is the best TC forecast model
- reduce 'big' errors
- do big errors happen within a storm or by storm?
- Forecast Error (FE)
- is not the same as PE or IE
- must be related to the wind field (the forecast) and particularly the extent of 34 kt winds represented by the Radius of 34 kt winds (R34)
- conceptually FE=f(PE,IE,R34)
- in the early years (60-80s) Charlie Neumann defined FE=PE
- or FE=a*PE + b*IE ; a=1.0 and b=0.0
- New FE=f(PE,IE)
- only require the forecaster (human or model) to predict position and intensity use best track (BT) R34
- define error IKE (integrated kinetic energy, Powell...) as symmetric difference of two (intersecting) circles of R34
- symmetric R34
- statistical relationship between Vmax and Rmax for forecast & BT
- use of this simplified IKE represents a lower limit in FE
- How
- demonstrate how 'large' errors contribute to the season/basin mean
- for every one 'bad' forecast it takes 5-10 'good' forecasts to compensate
- one storm can dominate the mean at the medium/long range
- demonstrate how storms contribute to the mean PE/IE and why a model failure for a single storm can 'ruin' the seasonal mean
- analyze 'large' model errors for both PE and FE
- are large PE errors always large FE?
- how do large FE compare to PE
