Purpose - To facilitate use of a robust, independent and global common benchmarking and assessment system for temperature data-product creation methodologies to aid product intercomparison and uncertainty quantification:
http://www.surfacetemperatures.org/benchmarking-and-assessment-working-group
Posting and comments are open for constructive advice and ideas.
Thursday, 10 February 2011
Assessing the Benchmarks
When we eventually design the benchmarks we need to keep in mind our assessment aims and outputs. They need to be designed such that it is easy to obtain useful and meaningful measures of skill. I've been chatting to Ian Jolliffe about the use of simple forecast verification techniques (break present and break detected = very good, break present and break not detected = bad, no break and break detected = very bad, no break and no break detected = good) including ROC plots. I'll go into more detail on this at some point soon but just thought I would throw the thought out there in case anyone has any bright ideas/thoughts.
4 comments:
My favourite error assessment measures are related to what the user expects to be able to do with homogenised data: study trends, decadal variability and relations between variables. Thus a basic measure would be the root mean square error (rmse; after subtracting the means of the truth and the homogenised time series as homogenisation does not aim at reconstructing the mean values better), or the rmse of the smoothed time series (up to decal scales), and finally the accuracy of the parameters of a (linear) trend estimate.
Contingency scores (hit rates, false alarm rates, Relative operating characteristic (ROC), Equitable threat score (ETS), Heidke skill score (HSS), etc.) are especially useful to understand why a homogenisation algorithm performs well or not. The developer of an algorithm that is very good in detection, but not in the rmse or trends, should have a look at the correction method or the way the iterations are implemented.
My take: if we know the truth of the simulated data, then we know the mean (in time) of the data we've simulated. After we homogenize the series, one has an estimated mean (in time). So simply compute the correlation coefficient between the estimated mean and the true mean. That leaves a number between -1 and +1 that is easily interpreted and applies to any changepoint configuration we can concoct. The game would then be who can get the correlation closest to +1.
I have a slight preference for the root mean square error (after setting the mean of both time series to zero) above the correlation coefficient. The latter measure would not notice if there were problems with the variance of the homogenised time series. For monthly data this is a minor issue, its variance is likely not much wrong.
However, for daily data reconstructing the right variance is important and we should therefore also estimate the skill of the homogenisation algorithms in this respect. I agree with Robert Lund and his previous comment
http://surftempbenchmarking.blogspot.com/2011/01/homogenization-aspects-that-scare-me.html?showComment=1297960394885#c1923189940425223935”>
that it is tricky to correct changes in the distribution and that this can easily lead to destroying the data and not to improving it. The papers on correction methods for daily climate data show that you only improve the data if the spatial correlations are strong enough. As far as I know automatic methods that gradually make a transition from correcting the full distribution for well cross-correlated data to only the mean for less well corrected stations do not exist yet (it may be possible to improve SPLIDHOM (Mestre et al, 2011) to be able to do so).
We still have to work on correcting the distribution of daily data, simply because many people work with this data because they are interested in the distribution. People only interested in changes in the mean (e.g. trends in large scale climate) are best advised to use monthly data: there is much more monthly data (digitised), the amount of the data is less, its homogenisation is more reliable. Daily data with a reliable distribution is needed to study whether there are changes in the variability of the weather or whether there are trends in weather extremes. Does anyone know how important a homogeneous distribution is for downscaling, to study physical relations between variables or for assimilation of climate data? Would mean-adjusted daily data be sufficient for such applications?
Concerning ROC, take into account that detection is not always (if ever :-) ) good in position : changes are often detected but at +/- 2 points.
Post a Comment