Notes for Team Validation:
Use the existing benchmarks and validation to look at which methods are more or less useful. Importance of looking at both ability to recreate the 'truth' and to detect the different types of breaks so that algorithm creators can get something positive about this – what exactly is causing problems for the algorithms? (station density, break frequency, break magnitude, background trend, seasonal cycle, natural variability, missing data, breaks near end-points etc.).
Reference period – this should be the most recent homogeneous subperiod. This is a problem, especially for algorithms doing seasonal shifts, when the last breakpoint is very close to the end of the record. However, this could be a real break location and so should not deliberately be avoided. Assessment should be aware of this though – algorithms could be penalised by this because they would not be able to model the seasonality effectively but assessments may look like the algorithm is failing because of the types of breaks or another complicating feature that was added – importance of useful assessment.
I think that false alarm rates are very important. I would rather a conservative and low false alarm rate than one that gets a higher number of breaks but adds a lot of error too. The false alarm rate should take into account the impact of incorrectly detecting a break given the adjustment applied. A detected break with a negligible adjustment applied is not so bad.
RMSE error seems to be a simple and useful metric – to root or not to root though?
analog-error-worlds minus analog-known-worlds = FULLRMSE
adjusted analog-error-worlds minus analog-known-worlds = REDUCEDRMSE
REDUCEDRMSE should be less than FULLRMSE if the algorithm is improving the network
Watch out for temporal variation in contingency scores – fewer breaks detected near the end of series?
Validating on annual verses monthly (or daily) – should validate on the highest resolution that will be used – so monthly I would say. This will penalise against flat adjustments but we know that inhomogeneities are not flat changes – this means that the errors added MUST be as realistic as possible.
How to calculate True negatives for the contingency scores?
Ensemble approach – hopefully this approach will be growing in popularity and so we need to be able to cope with this. An argument for keeping validation simple.
Some algorithms are trying to adjust more than just the mean, some of the higher order moments. Do we know enough to be able to add in errors in this way? Can we assess this fairly?
10 comments:
IJ: It’s easiest to comment on Kate’s comments by pasting them in again, and interleaving my own, as below. Before that, a general comment is that it is eminently sensible to start with what COST/HOME do and then decide what parts we want to use ourselves, and what we might like to add. Victor is clearly the man who knows exactly what has been done, and who has already thought carefully about the choices. He circulated a document ‘Benchmarking monthly homogenisation algorithms’ a few months back. My understanding (which could be wrong) from reading this is that the following were used:
1. A centred version of RMSE, though I’m not sure I understand exactly what centring has been done.
2. RMSE for regression coefficients to examine how well trend is reproduced in homogenised data.
3. Hit rate, false alarm rate, Heidke Skill Score (HSS) and a modified version of HSS.
Is this all?
What follows is in two parts, because when I tried posting it all I was told it was too long.
KW: Use the existing benchmarks and validation to look at which methods are
more or less useful. Importance of looking at both ability to recreate
the 'truth' and to detect the different types of breaks so that
algorithm creators can get something positive about this - what exactly
is causing problems for the algorithms? (station density, break
frequency, break magnitude, background trend, seasonal cycle, natural
variability, missing data, breaks near end-points etc.).
IJ: I think this a job for Team Corruption. The design of the analog-error-worlds will determine whether these things can be disentangled or are confounded.
KW: Reference period - this should be the most recent homogeneous subperiod.
IJ: I’m not sure what is meant by the ‘reference period’ or what role it plays.
KW: This is a problem, especially for algorithms doing seasonal shifts, when
the last breakpoint is very close to the end of the record. However,
this could be a real break location and so should not deliberately be
avoided. Assessment should be aware of this though - algorithms could be
penalised by this because they would not be able to model the
seasonality effectively but assessments may look like the algorithm is
failing because of the types of breaks or another complicating feature
that was added - importance of useful assessment.
IJ: Second installment.
KW: I think that false alarm rates are very important. I would rather a
conservative and low false alarm rate than one that gets a higher number
of breaks but adds a lot of error too. The false alarm rate should take
into account the impact of incorrectly detecting a break given the
adjustment applied. A detected break with a negligible adjustment
applied is not so bad.
IJ: This is where value/cost/loss could come in, but because it will be different for different users, it would be difficult to come up with a widely accepted cost/loss/value trade-off for the different types of ‘mistake’.
KW: RMSE error seems to be a simple and useful metric - to root or not to
root though?
analog-error-worlds minus analog-known-worlds = FULLRMSE
adjusted analog-error-worlds minus analog-known-worlds = REDUCEDRMSE
REDUCEDRMSE should be less than FULLRMSE if the algorithm is
improving the network
IJ: MSE also has the advantage of being decomposable into various contributions. Its disadvantage is its sensitivity to outliers. MAE or MAPE are possible alternatives
KW: Watch out for temporal variation in contingency scores - fewer breaks
detected near the end of series?
IJ: It should be possible to test, using Poisson assumptions, whether there are significantly fewer breaks detected near the end of a series.
KW: Validating on annual verses monthly (or daily) - should validate on the
highest resolution that will be used - so monthly I would say. This will
penalise against flat adjustments but we know that inhomogeneities are
not flat changes - this means that the errors added MUST be as realistic
as possible.
KW: How to calculate True negatives for the contingency scores?
IJ: Yes, needs thought about the definition – perhaps linked to how close together breaks can be. What did COST/HOME do?
KW: Ensemble approach - hopefully this approach will be growing in
popularity and so we need to be able to cope with this. An argument for
keeping validation simple.
IJ: Second installment.
KW: Some algorithms are trying to adjust more than just the mean, some of
the higher order moments. Do we know enough to be able to add in errors
in this way? Can we assess this fairly?
IJ: I’m not sure what is meant by ‘adjust more than just the mean’. I guess I need to read some appropriate documentation.
Why did we use the centred normal root mean square error (RMSE) in HOME? Imagine there is a station with 100 years of data with one break. There was a move to another location 10 years ago. In the first 90 years the mean temperature was 15 degrees in the last 10 years it was 18 degrees.
The homogenized time series may have an average of 15 degrees (for people who took the longest part of the data as their reference) or 18 degrees (for people who took the last period as reference, which makes it easier to continue a time series with new measurements.)
Both results are fine, none should be penalized. A normal RMSE would do so. The true data would have mean of 15 or 18 degrees, arbitrarily depending on how the break was implemented.
Therefore, we decided to subtract the means of the true data and of the homogenized data, to centre the time series, before we computed the RMSE (centred RMSE; CRMSE).
For an intercomparison study, this is a reasonable solution, but it is not perfect for three reasons. Firstly, if you look at the temporal behaviour of the CMRSE, you will notice that its lowest values are in the middle of the time series. Some stations will have an upward or downward trend in their error, by centring the time series such an error is partially removed. Secondly, you have the same problem with the seasonal cycle. This first period could have another seasonal cycle as the last one. You could again remove the annual cycle by centring all monthly time series, but then you would no evaluate the reconstruction of the seasonal cycle at all. While you can say that homogenization is about temporal consistency and the absolute level (mean) of a time series is not (very) important, you cannot say this as easily about the seasonal cycle, which is climatologically important. Thirdly, the absolute level and location of the measurement is not completely irrelevant. If the change in location is too large, one typically does not merge two stations into one time series, but rather keeps them separate in the database. What is too large is somewhat arbitrary. If you would like to interpolate between stations, or would like to combine the station dataset with data on orography or land use, you may want to know which of the two locations was used as reference.
These problems can be avoided by stating in advance what will be the reference period. Then a normal RMSE can be used to evaluate the results in a way that is fair to all contributions. A good choice would be to take that last homogeneous subperiod (HSP; in the true data) as reference. This is how most homogenization algorithms currently work. In this case, we would need an additional rule on what to do when the last HSP is short because then the seasonal cycle cannot be estimated with confidence and it would not be clear whether the last period is a HSP or just and outlier. Thus we could state that if, e.g., the last HSP is less than 5 years, its values will be ignored and the second last HSP is used as reference. Or we could decide that in such cases, this station will not be included in the computation of the RMSE.
Alternatively, we could tell the homogenizers, which period (year) will be used as reference. This it would have the advantage that you could select a relatively long HSP in the recent past as reference. A long period would be good as it makes the estimates of the mean including seasonal cycle more reliable, which would make it easier to see subtler difference between the contributions. This would make the method more complicated for the homogenizers (because it is new and because the period would be different for every station). It would be good to stay in the recent past as data from the early instrumental period has more measurement problems. Taking the last period includes more data from automatic weather stations, which are typically seen as being inferior. Thus the longest HSP between 1900 and 1980 could be a definition for the reference period and we would have to state this period (or one year in it) for every station in our ISTI benchmark.
Kate Willett wrote: "I think that false alarm rates are very important. I would rather a conservative and low false alarm rate than one that gets a higher number of breaks but adds a lot of error too. The false alarm rate should take into account the impact of incorrectly detecting a break given the adjustment applied. A detected break with a negligible adjustment applied is not so bad."
I think that false alarm rates are not important. I would rather be liberal and aim to achieve a low root mean square error or low uncertainty in the trends. Those are the things a climatologist needs.
There is always a trade of between false alarm rate and hit rate. The more conservative you are, the smaller you would like your false alarm rate to be, the lower will also be your hit rate. A climatologist is not interested in our false alarm rates or hit rates, (s)he just wants to have the best possible data. Thus the best false alarm rate is not a low one, but the one that optimizes the homogeneity of the data.
The most used significance level, 5 %, is already very very low. I presume science chose this low level, because we do not want the papers to write every 20th experiment that the speed of light can be exceeded. For smaller less newsworthy hypothesis, e.g. whether there is a break in 1950 in the temperature record of Groix, a higher level may also be acceptable as long as the data quality improves on average.
In case you missed a break, you may feel passively involved; you did not cause the break. In case you add a break where there is not you may feel actively involved; the data has a break because of your actions. In both cases there is a break and climatologist has a problem. The climatologist does not care what the reason is. Thus in the end, this problem is closer to the classical ethical problem, are your just as responsible for what you did not do as for what you did do? In principle, I would answer this question with a clear yes. While, I do realise that you cannot life this way; someone who may never kill someone, will not be active all day to prevent starving poor people far away from dying, but maybe one should
Thus I understand why no one wants to be responsible for adding inhomogeneities to the data by accident, but maybe it is more important to find more inhomogeneities that are there. Maybe we should just care about the (average) quality of the end results.
Ian Jolliffe wrote: “My understanding (which could be wrong) from reading this is that the following were used (by the Cost Action HOME, red):
1. A centred version of RMSE, …
2. RMSE for regression coefficients to examine how well trend is reproduced in homogenised data.
3. Hit rate, false alarm rate, Heidke Skill Score (HSS) and a modified version of HSS.
Is this all?”
Yes, in principle. The centred RMSE (CRMSE) was computed on monthly and yearly data as well as on smoothed data, the later can also be seen as a validation for nonlinear trends. Furthermore, the CRMSE and the RMSE of trend estimates were computed for the station data, as well as for the network mean signal.
The first paper main aim was to introduce the benchmark dataset, it only made a first cursory analysis of the results. Much more detailed analysis is possible, some ideas are mentioned at the end of the paper: “How well the gradual local trends are removed by homogenization would warrant a dedicated study, as well. …Users may for instance be interested in the annual cycle, the cross-correlations between stations, as well as secular trends for individual months, (interannual) variability, intermittence and long range dependence (Rust et al., 2008).“
For an understanding of the algorithms, these error metrics may be combined with contingency scores. Not only the simple ones above, but also, as Kate suggested, taking into account the uncertainty in the break date and size.
For our global dataset, it would probably be interesting to also study the results for subregions. The errors will likely be much smaller in Europe, due to its denser network, as in the Southern Hemisphere. A climatologist working on a certain regions, would be happy for some guidance on this specific region.
We will also be introducing network wide inhomogeneities, which is possible for the ISTI as the global dataset contains many different networks. It would be interesting to study whether the quality of the data depends on the distance from the boundary of the network. Maybe not all algorithms would notice the breaks in station in the centre of the network.
I have at last looked at Victor's comments, and I'm afraid it has revealed just how little I understand regarding the structure of the 'true', corrupted and homogenised data. My guess is that I need to see equations and numerical examples (and/or have face-to-face discussions) before I feel confident that I know what we are dealing with and hence how to proceed.
Hi, I think looking at what COST HOME have done is the best idea here. Victor - sorry I think you've told me this before - where can we find the results of the Benchmarking? I've found the benchmarks themselves?
I still like the idea of having two components to the analysis: how close the adjusted data are to their original homogeneous beginnings (at station and region level); how well the algorithm does at detecting and adjusting for specific breaks? The latter would be akin to a hit rate/false alarm rate. This is important because its useful to know if an algorithm is better/worse at detecting breaks of a particular sign, within a particular season, under El Nino/La Nina modes of variability, in the presence/absence of a strong background trend etc. If Team Corruption can add specific types of breaks (station moves, instrument changes, screen changes, observation time change) then having a hit rate/false alarm rate may be quite revealing as to which types of breaks are easier to detect/adjust for. Of course this depends how realistically we can recreate these break types.
Ian - I'm happy to get the HOME benchmark results/data into some useful format for you if that's any help.
Kate
I'm not sure what would be a useful format. I don't anticipate having the facilities to do any analysis of the data/results. What I would like is sufficient detail about them to understand how they were produced.
Hi Kate and Ian,
Sorry, if I write in a way that is not very accessible for people who have not read our paper. I do my best, but it is hard to estimate what you need to explain and what not and I also try to keep the posts and comments short, which is already often not very successful. :-)
The results?
Do you mean the returned homogenized data and the raw and original data?
Or do you mean our article with a first analysis of the results on the homepage of Climate of the Past Discussions. (I just got the proofs, in January the final version will be published in Climate of the Past.
If the article does not have sufficient detail about how the benchmark data was produced, you can find more information in a report.
The two analysis components is very good. How close the adjusted data is to the original is important for the climatologist using homogenized data. Combining this with information on hit rates (etc) you can try to understand which components of the homogenization algorithms works well.
I also like the idea to try to insert different types of inhomogeneities and later on make the analysis of the results for each of these types.
Thanks Victor - I think you've pointed me in the direction of what I wanted. Let me know as soon as the paper is published - we should point to it from our website.
Post a Comment