Here are my first set of thoughts on what our team could and should
be doing. There may be things that I've completely overlooked. Please
send any comments you have on omissions, or on any of my thoughts, as
soon as you think of them.
I see three things that fall within our remit:
1. Identify any experts that we would like to join us and issue invitations.
2. Identify which validation/verification techniques we should use.
3. Find or write software/code to implement the chosen techniques.
Let's say a bit more on each of these:
1. Until we have made progress on 2., it is difficult to decide who
best to invite. We could, of course, ask someone with general
verification knowledge rather than someone specialising in the types
of data format we identify in 2. Please email any suggestions to me.
I can think of several, but no one of them stands out as first choice.
At this stage there may not be much for them to get their teeth into.
2. We will not be certain what the data format will be until Team
Corruption have made decisions. I guess this will not be finalised
until sometime in 2012, so we could argue that we can do nothing
until then. However, I'm sure we can make some educated guesses
as to what will need to be validated. If there are some things we can
be fairly certain of, we can make decisions for those formats, but
not waste time considering scenarios that might not be used.
A quick look at some possibilities/questions:
was a changepoint found Yes/No?
was the nature of the changepoint correctly identified - this could
again be Yes/No or it could quantify how closely the magnitude
of a change was estimated. Different types of change would need
different validation methodology.
a key question is whether validation will be done station-by-station
and the results simply added, or whether an attempt will be made
to assess how well the spatial pattern of the 'corrected' data match
the 'true' data? The answer to this would determine whether we
want to bring on board an expert in spatial verification.
will we want to compare the distribution of 'corrected' data with
that of the true 'data', as well as looking at how well individual
corrected and true data sets match?
some ideas are given in Section 2.5 of
whitepaper_Benchmarking_Jun2011_v2.pdf
(available on the group website) and also in Section 5 of the COST
(HOME) paper circulated by Victor on July 5th.
3. This stage needs some serious investment of time, and computing
expertise, and is not something I can contribute much to. We could
certainly do with an expert on this side of things. It can't be started
until we are well advanced with 2. but we could start thinking about
who/how will do this.
Ian
be doing. There may be things that I've completely overlooked. Please
send any comments you have on omissions, or on any of my thoughts, as
soon as you think of them.
I see three things that fall within our remit:
1. Identify any experts that we would like to join us and issue invitations.
2. Identify which validation/verification techniques we should use.
3. Find or write software/code to implement the chosen techniques.
Let's say a bit more on each of these:
1. Until we have made progress on 2., it is difficult to decide who
best to invite. We could, of course, ask someone with general
verification knowledge rather than someone specialising in the types
of data format we identify in 2. Please email any suggestions to me.
I can think of several, but no one of them stands out as first choice.
At this stage there may not be much for them to get their teeth into.
2. We will not be certain what the data format will be until Team
Corruption have made decisions. I guess this will not be finalised
until sometime in 2012, so we could argue that we can do nothing
until then. However, I'm sure we can make some educated guesses
as to what will need to be validated. If there are some things we can
be fairly certain of, we can make decisions for those formats, but
not waste time considering scenarios that might not be used.
A quick look at some possibilities/questions:
was a changepoint found Yes/No?
was the nature of the changepoint correctly identified - this could
again be Yes/No or it could quantify how closely the magnitude
of a change was estimated. Different types of change would need
different validation methodology.
a key question is whether validation will be done station-by-station
and the results simply added, or whether an attempt will be made
to assess how well the spatial pattern of the 'corrected' data match
the 'true' data? The answer to this would determine whether we
want to bring on board an expert in spatial verification.
will we want to compare the distribution of 'corrected' data with
that of the true 'data', as well as looking at how well individual
corrected and true data sets match?
some ideas are given in Section 2.5 of
whitepaper_Benchmarking_Jun2011_v2.pdf
(available on the group website) and also in Section 5 of the COST
(HOME) paper circulated by Victor on July 5th.
3. This stage needs some serious investment of time, and computing
expertise, and is not something I can contribute much to. We could
certainly do with an expert on this side of things. It can't be started
until we are well advanced with 2. but we could start thinking about
who/how will do this.
Ian