Thursday, 7 November 2013

Summary of Regional Inhomogeneities

A few months ago the Benchmarking group made a call to the homogenisation community to submit any times/characters of known inhomogeneities occurring in different regions.

Thanks to the many contributors we now have an overview for a number of countries:
Spain
Poland
Canada
Switzerland
Australia
Tropics
Italy
Central Europe
Russia
Slovenia
Netherlands
UK
Austria
USA

There is always room for more. If you come across any useful information or would just like to see what is there so far please go to:

https://docs.google.com/spreadsheet/ccc?key=0Al6ocsUAaINSdHpTREJzVkRZUTdfVjNPRlh0Q1V3WUE&usp=sharing#gid=0

This is an online editable document so please do add info. A static version will be scraped once a month.




Kate










ISTI/Benchmarking talk at Reading University Meteorology Department lunchtime seminar

I was invited to present the work of the ISTI at Reading University Meteorology Department.

The talk covered the aims and workings of the ISTI: to facilitate robust climate analysis of land surface temperature.

1) Provide a version controlled, comprehensive, traceable to known origin, openly shared databank of surface temperature data with digitally attached metadata.

2) Set up an international standard for benchmark testing homogenisation algorithms and other methodological choices for building climate data-products to help quantify homogenisation and methodological uncertainty and aid product selection and methodological advancement.

3) Provide a portal for all ISTI related products with user tools to help inform choice of product, visualisation and intercomparison of products.

I then focussed on the benchmarking side of things to explain why this is needed and how we are going about it.

The slides for the talk are available here: The ISTI: Dragging the land surface temperature data kicking and screaming into the 21st Century

A few of the interesting questions that I remember:

Will the ISTI databank include non-standard station data?
Yes, there are plans to incorporate data from amateur stations. This will be flagged and prioritised appropriately.

How do you deal with uncertainties between the true shaded air temperature verses the biases created by poor ventilation or insolation issues in screen temperatures?
This is a big problem as we need to quantify changes/impacts on the smaller, more local scale. By making as much data available as possible, in addition to metadata, the ISTI can help address this problem. Having a higher density of stations increases the statistical power in detection of errors. Making it easier to build surface temperature products enables more people to tackle the problem and apply a wider spread of methodological choices. This may help to constrain the uncertainty to come extent.

Can you provide a list of stations used to make up each gridbox in gridded products?
This is something that we would encourage anyone building data-products to do. Ideally they would list the stations and which stage and version they come from. The data-portal will have to be able to cope with hosting ancillary information pertaining to the data-products in an easily searchable and usable manner.

Can other people make their own error worlds/synthetic data?
Yes - all of our code will be written in R and published on the website. The idea is that others can use the code and tweak various parameters to make their own clean synthetic stations and add errors as they wish.

Monday, 19 August 2013

Benchmarking Workshop Agenda and Report

The workshop agenda and full report can be found here: https://sites.google.com/a/surfacetemperatures.org/home/benchmarking-and-assessment-working-group#Minutes

Below is the executive summary.

1st – 3rd July 2013 Benchmarking Working Group Workshop Report Executive Summary National Climatic Data Center (NCDC) of the National Oceanic and Atmospheric Administration (NOAA), Asheville, NC, USA

Attended in person:
Kate Willett (UK), Matt Menne (USA), Claude Williams (USA), Robert Lund (USA), Enric Aguilar (Spain), Colin Gallagher (USA), Zeke Hausfather (USA), Peter Thorne (USA), Jared Rennie (USA)
 

Attended by phone:
Ian Jolliffe (UK), Lisa Alexander (Australia), Stefan Brönniman (Switzerland), Lucie A. Vincent (Canada), Victor Venema (Germany), Renate Auchmann (Switzerland), Thordis Thorarinsdottir (Norway), Robert Dunn (UK), David Parker (UK)


A three day workshop was held to bring together some members of the ISTI Benchmarking working group with the aim of making significant progress towards the creation and dissemination of a homogenisation algorithm benchmark system. Specifically, we hoped to have: the method for creating the analog-clean-worlds finalised; the error-model worlds defined and a plan of how to develop these; and the concepts for assessment finalised including a decision on what data/statistics to ask users to return. This was an ambitious plan for three days with numerous issues and big decisions still to be tackled.

The complexity of much of the discussion throughout the three days really highlighted the value of this face-to-face meeting. It was important to take time to ensure that everyone understood and had come to the same conclusion. This was aided by whiteboard illustrations and software exploration, which would not have been possible over a teleconference.

In overview, we made significant progress in terms of developing and converging on concepts and important decisions. We did not complete the work of Team Creation as hoped, but necessary exploration of the existing methods was undertaken revealing significant weaknesses and ideas for new avenues to explore have been found.

The blind and open error-worlds concepts are 95% complete and progress was made on the specifics of the changepoint statistics for each world. Important decisions were also made regarding missing data, length of record and changepoint location frequency. Seasonal cycles were discussed at length and more research has been actioned. A significant first go was made at designing a build methodology for the error-models with some coding examples worked through and different probability distributions explored.

We converged on what we would like to receive from benchmark users for the assessment and worked through some examples of aggregating station results over regions. We will assess both retrieval of trends and climate characteristics in addition to ability to detect changepoints. Contingency tables of some form will also be used. We also hope to have some online or assessment software available so that users can make their own assessment of the open worlds and past versions of benchmarks. We plan to collaborate with the VALUE downscaling validation project where possible.

From an intense three days all participants and teleconference participants gained a better understanding of what we're trying to achieve and how we are going to get there. This was a highly valuable three days, not least through its effect of focussing our attention prior to the meeting and motivating further collaborative work after the meeting. Two new members have agreed to join the effort and their expertise is a fantastic contribution to the project.

Specifically, Kate and Robert are to work on their respective methods for Team Creation, utilising GCM data and the vector autoregressive method. This will result in a publication describing the methodology. We aim to finalise this work in August.

Follow on teleconferences, Team Corruption will focus on completing the distribution specifications and building the probability model to allocate station changepoints. This work is planned for completion by October 2013. Release of the benchmarks is scheduled for November 2013.

Team Validation will continue to develop the specific assessment tests and work these into a software package that can be easily implemented. This work is hoped to be completed by December 2013, but there is more time available as assessment will take place at least 1 year after benchmark release.


Monday, 13 May 2013

Call for regional inhomogeneity info

To create realistic benchmarks we would like to reproduce times and locations of known sources of inhomogeneity as best we can. Please can you help us. If you know of any regional/countrywide changes to the observing system over time please can you list them here or point us to some documentation/reference. Any information is valuable - even if its quite vague.

Ideally we'd like to know:

WHEN - specific date or month or year or even decade etc.
WHERE - a region, a country, an international GTS/WMO change etc.
WHAT - a change in shelter, thermometer type, automation, observing time/practice etc.
HOW - are there any estimates of the size/direction/nature of the effect of this change?

Please post here and encourage others to do so. We then hope to reward you with some realistic error-worlds to play with.

Kate

Sunday, 15 January 2012

Mailing list on homogenisation of climate data

There is now a homogenisation mailing list, which aims to strengthen the communication and co-operation in the scientific community on homogenenisation. Suggested uses of the list are announcements on:

  • Conferences, workshops, etc.
  • Important papers
  • Discussions
  • Job opportunities
  • New projects started
  • Requests for data, information, and cooperation partners
  • ...
For more information (on how to subscribe), please to go my page on the homogenisation mailing list.

Tuesday, 10 January 2012

New article: Benchmarking homogenization algorithms for monthly data

The main paper of the COST Action HOME on homogenization of climate data has been published today in Climate of the Past. This paper could be seen as a pre-study for the benchmarking in the international surface temperature initiative (ISTI)

Inhomogeneities

To study climatic variability the original observations are indispensable, but not directly usable. Next to real climate signals they may also contain non-climatic changes. Corrections to the data are needed to remove these non-climatic influences, this is called homogenisation. The best known non-climatic change is the urban heat island effect. The temperature in cities can be warmer than on the surrounding country side, especially at night. Thus as cities grow, one may expect that temperatures measured in cities become higher. On the other hand, many stations have been relocated from cities to nearby, typically cooler, airports. Other non-climatic changes can be caused by changes in measurement methods. Meteorological instruments are typically installed in a screen to protect them from direct sun and wetting. In the 19th century it was common to use a metal screen on a North facing wall. However, the building may warm the screen leading to higher temperature measurements. When this problem was realised the so-called Stevenson screen was introduced, typically installed in gardens, away from buildings. This is still the most typical weather screen with its typical double-louvre door and walls. Nowadays automatic weather stations, which reduce labor costs, are becoming more common; they protect the thermometer by a number of white plastic cones. This necessitated changes from manually recorded liquid and glass thermometers to automated electrical resistance thermometers, which reduces the recorded temperature values.



One way to study the influence of changes in measurement techniques is by making simultaneous measurements with historical and current instruments, procedures or screens. This picture shows three meteorological shelters next to each other in Murcia (Spain). The rightmost shelter is a replica of the Montsouri screen, in use in Spain and many European countries in the late 19th century and early 20th century. In the middle, Stevenson screen equipped with automatic sensors. Leftmost, Stevenson screen equipped with conventional meteorological instruments.
Picture: Project SCREEN, Center for Climate Change, Universitat Rovira i Virgili, Spain.


A further example for a change in the measurement method is that the precipitation amounts observed in the early instrumental period (about before 1900) are biased and are 10% lower than nowadays because the measurements were often made on a roof. At the time, instruments were installed on rooftops to ensure that the instrument is never shielded from the rain, but it was found later that due to the turbulent flow of the wind on roofs, some rain droplets and especially snow flakes did not fall into the opening. Consequently measurements are nowadays performed closer to the ground.

Homogenization

To reliably study the real development of the climate, non-climatic changes have to be removed. For this the small difference of one station to its direct neighbours are utilized. In this way non-climatic changes (shelter and instrument changes or station moves usually) in a single stations can be more clearly seen as in the record of one station by itself due to the strong natural climatic variability. This method does not work when changes are applied to a whole country’s network. Such extensive changes are less problematic, however, because are typically well documented.




Meteorological window suggested by Italian Central Office for Meteorology and Climate in 1879 (Tacchini, 1879). In the last decades of the 19th century most of Italian observations were performed in urban environments, in screens located outside a north-facing window of the highest floor of a “meteorological tower”. The purpose of using such towers was to perform observations above the level of the roofs of the surrounding buildings. Picture: Michele Brunetti, ISAC-CNR, Bologna, Italy.

Benchmarking

To study the performance of the various homogenisation methods, the COST Action HOME has performed a test with artificial climate data. The advantage of artificial data is that the non-climatic changes are known to those who created the data. The artificial data used mimics climatic networks and their data problems with unprecedented realism. For this we have used the IAAFT algorithm, which can generate non-Gaussian data with arbitrary temporal variability and cross-correlations between the stations. The artificial data may have a warming, a cooling or no trend, to ensure objective testing of the methods. The main novelty is that the test was blind. In other words, while homogenising the data the scientists did not know which station contained which non-climatic problem. The artificial data were generated and the analysis of results was performed by independent researchers, who did not homogenise the data themselves. Consequently, the COST Action is sure that the results are an honest appraisal of the true power of homogenisation algorithms.

Some people remaining sceptical of climate change claim that adjustments applied to the data by climatologists, to correct for the issues described above, lead to overestimates of global warming. The results clearly show that homogenisation improves the quality of temperature records and makes the estimate of climatic trends more accurate.



The photo on the right shows an open shelter for meteorological instruments at the edge of the school square of the primary school of La Rochelle, in 1910. La Rochelle is a coastal city in western France and a seaport on the Bay of Biscay. On the left one sees the current situation, a Stevenson-like screen located closer to the ocean, along the Atlantic shore, in place named "Le bout blanc". Behind the fence you see the water of the port. Picture: Olivier Mestre, Meteo France, Toulouse, France.

Methodological advances in homogenization

In the past it was customary in homogenisation to compare a station with its neighbours by creating a reference time series from averaging over multiple neighbouring stations. Due to the averaging the influence of random non-climatic factors is strongly reduced. Thus if a jump was found in the difference time series of a station with its reference, the jump was assumed to be in the station, not in the reference, which was assumed to be homogeneous. In recent years climatologists and statisticians have worked on advanced statistical methods that do not need a homogeneous reference. The traditional methods reduced the influence of non-climatic factors on the temperature measurements, but the complex modern methods clearly improved the data much more. This finding could only be reached using the benchmark data simulating complete networks with realistic non-climatic problems. Thus now we can recommend with confidence that climatologists should use the new methods. These recommendations are, of course, not only based on the numerical results, but also on our mathematical understanding of the algorithms.

Open-access publishing

The scientific article with 31 authors describing this study has been published today in the journal Climate of the Past. This international journal is an open-access and an open-review journal of the European Geosciences Union. The articles of open-access journals can be freely read by anyone; the costs of publication are born by the authors. Open-access publishing makes it easier for researchers, also from poorer countries, to stay up to date and to participate in science. Also the general public can profit from open-access publishing as the access to the primary source can make the public debate on current scientific issues in newspapers and blogs more informed. Especially for this topic, we felt it was important that everyone can read the article. Next to many EGU journals, last week also the meteorological journal Tellus joined the open access movement.

Climate of the Past is also an open-review journal. This new way of reviewing scientific articles is public, everyone has the possibility to respond to the initial draft of the paper and everyone can read these comments as well as the comments of the official peer reviewers of the manuscript.

For more information

Venema, V., O. Mestre, E. Aguilar, I. Auer, J.A. Guijarro, P. Domonkos, G. Vertacnik, T. Szentimrey, P. Stepanek, P. Zahradnicek, J. Viarre, G. Müller-Westermeier, M. Lakatos, C.N. Williams, M. Menne, R. Lindau, D. Rasol, E. Rustemeier, K. Kolokythas, T. Marinova, L. Andresen, F. Acquaotta, S. Fratianni, S. Cheval, M. Klancar, M. Brunetti, Ch. Gruber, M. Prohom Duran, T. Likso, P. Esteban, Th. Brandsma. Benchmarking homogenization algorithms for monthly data. , Climate of the Past, 8, pp. 89-115, 2012.

If you would like to analyse the data used in this study, please go to this page for a link to the data as well as to documents that describe the dataset and data formats in detail.

The homepage of the Action HOME with amongst others a bibliography with most if not all articles on homogenization of climate networks.

If you are interested in homogenization, please send me an e-mail and I will put you on our email distribution list.

Friday, 6 January 2012

Benchmarking of USHCN

Cross-posted from the main blog. Please submit comments there.

Firstly, an up-front caveat, I am third (last) author on this paper.

Today the Journal of Geophysical Research has published a paper that applies the benchmarking and assessment principles of the surface temperature initiative to the USHCN dataset of land surface air temperatures.

Williams, C. N., Jr., M. J. Menne, and P. Thorne
Benchmarking the performance of pairwise homogenization of surface temperatures in the United States
J. Geophys. Res., doi:10.1029/2011JD016761, in press.  (behind a paywall - sorry)

The analysis takes the pairwise homogenization algorithm used to create the GHCN and USHCN products and does two things.

Firstly, it identifies a large number of decision points within the algorithm that do not have an absolute basis and allows these to vary. A good climate science analogy here is the perturbed climate model runs of the climateprediction.net project and other similar projects. These decision points were varied by random seeding of values to create a 100 member ensemble of solutions. This at least starts to explore the parametric uncertainties within the algorithm (and any interdependency's) and their implications for our understanding of the observed temperature record evolution. However, what it does not necessarily do is give us any better an idea as to what the true climate evolution may have been. Which brings us on to the second innovation and the focus of this post ...

Secondly, in addition to running on the observations these ensembles were run on a set of eight analogs to the USHCN network. These consisted of sets of data which directly mimicked the observational availability of the USHCN network itself through time. They were based upon climate model runs from a range of models and a range of forcing scenarios. This ensures some 'plausible' spatio-temporal coherency to the large-scale temperature fields. On top of these were super-imposed additional differences to mimic potential random and systematic influences of instrumental and operational artifacts. A set of distinct possibilities were explored across the eight worlds ranging from no systematic biases at all (highly improbable but a useful 'algorithm does no harm' test) through to a scenario where the network was bedevilled with very many largely small breaks with a sign-bias tendency - a situation which any algorithm would find hard to cope with. Unlike the real-world these analogs afford a luxury of knowing the true answer so that it is possible to actually benchmark and understand the fundamental algorithm performance and any limitations. Then it is possible to re-evaluate the real-world results afresh with these new insights gleaned from such realistic test-beds.

The results from the analogs were broadly encouraging. First and foremost when applied to the data with no breaks added virtually no adjustments were made and the impact on large-scale averaged timeseries and trends was so minuscule a magnifying glass would be required to tell the difference. So, in the implausible eventuality that the raw data are bias free the algorithm really would do no harm. For the other analogs the performance was mixed. Easier cases where breaks were bigger and metadata better it did better. Harder cases it fared worse. Where there was no overall sign bias in the applied breaks the ensemble was spread relatively evenly around the starting data. But where there was an overall bias in the raw data presented to the algorithm it consistently moved the overall data in the right direction but rarely far enough. The implication being that this uncertainty is effectively one-tailed. The chances of over-shooting the adjustments is substantially smaller than the chances of under-shooting the required adjustments.

So, what does the real-world look like when reassessed through this new understanding?

For minimum temperatures over the longest timescales the trends are spread around the raw data. But in the periods 1951 onwards and 1979 onwards it is spread distinctly either side of the raw data. Post 1951 trends in the raw data exhibit too little warming, and post-1979 too much. This is consistent with prior understanding of the biases of the change of time of observation (largely 1950s-1970s; spurious cooling) and move to MMTS sensors (early to mid-1980s and often associated with a microclimate relocation; spurious warming) on Tmin.

For maximum temperatures the ensemble consistently precludes the raw observations over all considered timescales. The raw data are almost certainly biased and show too little warming. Over all periods the ensemble of solutions show more warming than the raw data. Again, this is consistent with current understanding of the impact of time of observation biases (again a spurious cooling) and the transition to MMTS which unlike for minimum temperatures imparts a spurious cooling effect. The less than encouraging implication from the analogs, however, is that in the operational USHCN algorithm we are substantially more likely to be under-estimating the required adjustments, and hence rate of warming in maximum temperatures, than over-estimating it.

Is this the last word on the issue? Certainly not. As this was a first step along this path the analogs were perhaps not as sophisticated as would ideally be the case. And here we hope the benchmarking and assessment group can provide more realistic (and global) analogs later this year. Further, this considered solely parametric uncertainty - varying choices within one algorithm. The larger and more difficult uncertainty to understand is the structural uncertainty that would result from applying fundamentally distinct approaches to the same problem and allow a better exploration of the possible solution space. And here we have to look to you, readers, to develop new and novel approaches to homogenizing the data and submit them to the same raw data and analogs to allow consistent benchmarking and better understanding.

Finally, over coming weeks we will be hosting code, data, and metadata (including the analogs) online. I'll provide an update when it is all up there but given that its several Gb and requires fitting around other duties its not going to be instantaneous by any stretch.

Comments are welcome, but please remember that this is a strictly moderated blog and to follow the house rules.