Here are my first set of thoughts on what our team could and should
be doing. There may be things that I've completely overlooked. Please
send any comments you have on omissions, or on any of my thoughts, as
soon as you think of them.
I see three things that fall within our remit:
1. Identify any experts that we would like to join us and issue invitations.
2. Identify which validation/verification techniques we should use.
3. Find or write software/code to implement the chosen techniques.
Let's say a bit more on each of these:
1. Until we have made progress on 2., it is difficult to decide who
best to invite. We could, of course, ask someone with general
verification knowledge rather than someone specialising in the types
of data format we identify in 2. Please email any suggestions to me.
I can think of several, but no one of them stands out as first choice.
At this stage there may not be much for them to get their teeth into.
2. We will not be certain what the data format will be until Team
Corruption have made decisions. I guess this will not be finalised
until sometime in 2012, so we could argue that we can do nothing
until then. However, I'm sure we can make some educated guesses
as to what will need to be validated. If there are some things we can
be fairly certain of, we can make decisions for those formats, but
not waste time considering scenarios that might not be used.
A quick look at some possibilities/questions:
was a changepoint found Yes/No?
was the nature of the changepoint correctly identified - this could
again be Yes/No or it could quantify how closely the magnitude
of a change was estimated. Different types of change would need
different validation methodology.
a key question is whether validation will be done station-by-station
and the results simply added, or whether an attempt will be made
to assess how well the spatial pattern of the 'corrected' data match
the 'true' data? The answer to this would determine whether we
want to bring on board an expert in spatial verification.
will we want to compare the distribution of 'corrected' data with
that of the true 'data', as well as looking at how well individual
corrected and true data sets match?
some ideas are given in Section 2.5 of
whitepaper_Benchmarking_Jun2011_v2.pdf
(available on the group website) and also in Section 5 of the COST
(HOME) paper circulated by Victor on July 5th.
3. This stage needs some serious investment of time, and computing
expertise, and is not something I can contribute much to. We could
certainly do with an expert on this side of things. It can't be started
until we are well advanced with 2. but we could start thinking about
who/how will do this.
Ian
be doing. There may be things that I've completely overlooked. Please
send any comments you have on omissions, or on any of my thoughts, as
soon as you think of them.
I see three things that fall within our remit:
1. Identify any experts that we would like to join us and issue invitations.
2. Identify which validation/verification techniques we should use.
3. Find or write software/code to implement the chosen techniques.
Let's say a bit more on each of these:
1. Until we have made progress on 2., it is difficult to decide who
best to invite. We could, of course, ask someone with general
verification knowledge rather than someone specialising in the types
of data format we identify in 2. Please email any suggestions to me.
I can think of several, but no one of them stands out as first choice.
At this stage there may not be much for them to get their teeth into.
2. We will not be certain what the data format will be until Team
Corruption have made decisions. I guess this will not be finalised
until sometime in 2012, so we could argue that we can do nothing
until then. However, I'm sure we can make some educated guesses
as to what will need to be validated. If there are some things we can
be fairly certain of, we can make decisions for those formats, but
not waste time considering scenarios that might not be used.
A quick look at some possibilities/questions:
was a changepoint found Yes/No?
was the nature of the changepoint correctly identified - this could
again be Yes/No or it could quantify how closely the magnitude
of a change was estimated. Different types of change would need
different validation methodology.
a key question is whether validation will be done station-by-station
and the results simply added, or whether an attempt will be made
to assess how well the spatial pattern of the 'corrected' data match
the 'true' data? The answer to this would determine whether we
want to bring on board an expert in spatial verification.
will we want to compare the distribution of 'corrected' data with
that of the true 'data', as well as looking at how well individual
corrected and true data sets match?
some ideas are given in Section 2.5 of
whitepaper_Benchmarking_Jun2011_v2.pdf
(available on the group website) and also in Section 5 of the COST
(HOME) paper circulated by Victor on July 5th.
3. This stage needs some serious investment of time, and computing
expertise, and is not something I can contribute much to. We could
certainly do with an expert on this side of things. It can't be started
until we are well advanced with 2. but we could start thinking about
who/how will do this.
Ian
5 comments:
Response to point 2.
I think there is value in separating the ability of an algorithm to detect a break and the ability to correctly adjust the data to remove it.
Detection could be simply an: is a break found within +/- 3 months of the actual break location. It could go further to give greater weight to detections that get the correct type of break e.g., is it an abrupt shift? is it a gradual change? is it a seasonally varying break? I still like the idea of hits, misses, false alarms and nothings, perhaps weighted by which one is more damaging to the integrity of a station time series. In my mind detecting a break when there is not one is worse than not detecting one but only if the actual adjustment makes a difference to the dataset.
Validating the ability of algorithms to adjust is could be done at the station and regional level. Looking at the distribution may be a good idea - that would include an assessment of the mean, variance and other moments. Assessing the remaining background trend is also important. Can we measure whether the homogenisation process brings the data towards the truth or further away from it? Do the algorithms systematically under-correct, over-correct or make the data worse?
I have to admit I'm biased in my suggestions because I scribbled the white paper so it would be really good to hear from others who can think a little outside of the box.
To quantify the performance of a homogenization algorithm, break detection measures (and false alarm rates) are not very effective. If a break is detected, but almost no correction applied, this is not much of a problem. If in reality there are breaks in 1949 and 1951 and the algorithm puts one break in 1950, that is a good solution to the problem and should not be penalised much over a solution with two breaks.
If one algorithm finds many more small breaks, but makes more errors with the important large ones, this algorithm may perform well for detection, but lead to worse results as an algorithm that only finds large breaks, but does so perfectly. Given that homogenization can be viewed as a combinatorial problem, such differences are possible.
For understanding the homogenization algorithms break detection is important. Especially in combination with other measures (such as a RMSE or an error in the estimated trends). If an algorithm performs well on one of these tasks but not on the other, you get an idea where to search for improvements.
Algorithms may also perform break detection better or worse as a function of the size of the break, the seasonal cycle of the break, the properties of adjacent breaks, the accepted tolerance in the date of the break, etc. Such differences can again help in understanding the algorithms and give ideas for improvements.
I agree that we should not wait until the corruption team is finished. One way to be able to get started would be to analyse the HOME benchmark dataset. Then we would only need to rewrite the reading routines when the corruption team is finished.
Ian, could you explain in more detail what spatial verification is (and how it differs from station based verification)? Sounds interesting in case of homogenization. It is desirable to improve the spatial climatic patterns by homogenization.
That's a good point - that hit/false alarm rates for break detection are not an ideal metric if used in isolation. Some kind of weighting for getting the correct magnitude of the break is also important and I can see the complication between an algorithm that makes a single large adjustment when in fact two smaller breaks occurred. However, surely the preferred algorithm would be one that detects both breaks and makes sound adjustments. An algorithm detecting a single break may be satisfactory also, but not quite as good.
Having multiple components of the algorithm verification should help with this:
a) break location detection
b) break magnitude detection
c) ability to return the analog-error-world to its' homogenous analog-known-world in terms of the major statistics (mean, variance, trend etc.)
This would allow us to distinguish between:
- those algorithms that detect well and make sound adjustments
- those that detect fewer breaks but still manage to make realistic adjustments
- those that detect well but do not make sound adjustments
- and those that do not perform well in either criteria.
In reality I understand that it is unlikely to be so simple. Some algorithms will do well for some stations and not others and it may be difficult to make conclusive general statements about specific algorithms. This may be made easier by having a spatial component to this last bit (c) so that assessment can be informative to those using regional/hemispheric/global averages.
Using the HOME benchmarks as a starting point is a great idea and also learning from what did and didn't work well there. I just had a look at Tamas Szentimrey's presentation which had a nice way of assessing station/regional properties resulting from the homogenised datasets
(http://www.homogenisation.org/v_02_15/index.php?option=com_content&view=article&id=84%3Acost-benchmark-workshop-zurich&catid=46%3Areports&Itemid=55)
Thanks to Victor and Kate for their thoughts. This clearly isn't going to be straightforward. There are all sorts of things that could
be assessed, which could be done by purely mathematical/statistical measures or by bringing in considerations of cost/value, which Kate hints at. We need to strike a balance between giving a sufficiently detailed assessment to be as informative as possible, without going over the top and losing sight of the things that matter most - not seeing the wood for the trees.
Starting with HOME is a good idea - we can certainly use their validation ideas, as described in the paper circulated by Victor
in July. I'm not too sure what else Victor has in mind when he talks about 'analysing' the HOME benchmark dataset. What exactly would be done and by whom?
Regarding spatial verification, I'm retreating a bit on this. I mentioned it earlier because it has been one of the most active areas of development in verification in recent years. On reflection, not much of it may be relevant to what we want to do. The majority of the developments have been for detecting spatial 'objects' such as areas of precipitation.
It would probably be good to assess whether geographical patterns of temperatures are faithfully reproduced, as well as values at individual stations, but my guess is that this might be a secondary consideration. By definition if the pattern is wrong, then there must be errors at some of the individual stations, and so should be picked there. Perhaps the only potential use for spatial considerations to possibly explain, and hence help to suggest improvements, when problems are identified at individual stations.
There is still a lot that can be analysed on the benchmark data from the HOME Action. In this context, I was just thinking of setting up the validation system for the ISTI using a similar dataset, just to help us get going before the homogenized data of the ISTI is submitted.
Post a Comment