Archive | Science RSS for this section

The honourable work of data rescue

Gil Compo (centre) accepts the honourable mention oldWeather was awarded in the The 2013 International Data Rescue Award in the Geosciences, from organisers Kerstin Lehnert (IEDA, left) and Bethan Keall (Elsevier, right).

Gil Compo (centre) accepts the honourable mention for oldWeather from the 2013 International Data Rescue Award in the Geosciences, with organisers Kerstin Lehnert (IEDA, left) and Bethan Keall (Elsevier, right).

One of the fun parts of working as a scientist is going to conferences, and in the geosciences, conferences don’t come much bigger than AGU. The American Geophysical Union’s 46th annual Fall Meeting ran last week in San Francisco, and it brought together more than 22,000 scientists for a week of presentations, discussions, celebrations, and beer.

Our man at AGU this year was Gil Compo, and he represented oldWeather at an important side event: The prize ceremony for the 2013 International Data Rescue Award in the Geosciences. We didn’t quite win this prize (the winner was the excellent Nimbus Data Rescue Project), but the judges liked us a lot, and we were awarded an honourable mention. So well done to all the oldWeather participants on a further well-deserved honour, and thanks to the award sponsors and organisers.

Every scientist’s must-have accessory, at any large conference or meeting, is a poster: This is a large sheet of paper (typically A0, or about 4′ by 3′) covered with artistically arranged images and results from your project, which you attach to a wall or display board, and use as a visual aid. Kevin made an excellent poster for us, combining images from all aspects of the project. You can see it on display in the background of the photo above, and if you’d like your own copy, it’s on our resources page.

Scientific progress goes …

The oldWeather distributed supercomputer

Nobody succeeds alone, and that’s doubly true of oldWeather: not only are we legion in ourselves – a community of thousands working on logbook weather, but even as a project we are embedded in a community – we have friends and relations.

Our close relations, of course, are the other Zooniverse projects: That’s a diverse family – from the paterfamilias to the newest member, united by shared principles and the talents of the core team. But we also have more distant relatives. oldWeather is neither the first, nor the biggest, climate and weather citizen science project. (CPDN) turned ten this year, and they have a very different way of doing science.

Many of the experiments climate scientists would like to do are impossible in practice: What would happen to the weather, for example, if we were to induce artificial volcanoes as a way to cool the planet? To investigate these questions, we do simulations – we build computer models of the climate system and do the experiment in the model. We have learned an enormous amount by doing this, but it does take a lot of computer time. CPDN asks volunteers to let their desktop computers contribute to this work – most of the time we use only a small fraction of the power of our computers, so this work can be done entirely in your computer’s spare time – it does not interfere with your normal use.

CPDN is also part of a family: There are lots of volunteer computing projects sharing the infrastructure provided by the Berkeley Open Infrastructure for Network Computing (BOINC) and you can contribute to any you choose.

Several of the oldWeather community have doubled their efficiency by doing citizen science and volunteer computing simultaneously: while the people are reading logbooks, their computers are simulating the climate, or Neutron stars, or malaria, or the Milky Way, or … I’d like to congratulate the oldWeather BOINC group on their tremendous contribution both to oldWeather and to volunteer computing.

Too low, terrain!

Cockpit of a NOAA P3.

Cockpit of a NOAA P3.

oldWeather is telling us a great deal about how the present climate is different from that of 100 years ago, but to make maximum use of that information, we also want to know exactly how the present climate is behaving. This will help us link our observed changes in surface weather to the basic physics of the ocean and atmosphere. To learn about the present climate we collect a rich and detailed set of observations from research ships, aircraft, and satellites.

Last year, Kevin was out making such measurements from a ship, on a research cruise in the Bering Strait. This field season he’s back out there, but he’s gone up in the world. For some purposes ground level is too low, and satellites are too high, and to fill this gap NOAA have two research aircraft (affectionally known as ‘Kermit’ and ‘Miss Piggy’). Kevin’s group have got some time on one of them, they are trying to “quantify the air-ice-sea interactions and lower atmospheric structure in the marginal ice zone, with the ultimate goal of being able to infer how recent reductions in sea ice extent in autumn will impact the atmosphere“.

The research aircraft is complex and well-equipped: According to Kevin “The NOAA WP-3 is instrumented like ten satellites. So we are able to collect a vast array of data from deep oceanography with AXCTD and AXBT expendables, SST and surface microwave emission (wind/waves/ice), upward/downward radiation, up to 22 thousand feet where we deploy dropsondes from above the clouds to characterize the structure of the atmosphere. On a survey we collect flight level data continuously while deploying AX instruments about every six minutes.

To do all that effectively requires close cooperation between the crew of the aircraft and the scientists – that’s Kevin’s job. He’s sent back this video to give us a taste of what it’s like. It looks exciting – they spend a lot of time travelling at 200 knots, only 200 feet off the ground, much to the distress of the auto-pilot – but it’s hard work: one flight means 8-10 hours flight time + 2 hours for briefings before and after.

See more about this mission on the NOAA website.

Frost in the South

We’ve looked at the world from the top; this is the view from beneath: Antarctica in the centre, South America at top, South Africa right, Australia and New Zealand bottom left. Streamlines show near-surface wind, colours indicate temperature, dots mark rain and snow. All data are from the Met Office global analysis.

One reason why weather forecasting and climate research are hard is that the atmosphere is complicated: There’s a lot going on – all sorts of different motions and changes occurring simultaneously all over the world. So while it’s often useful to use simplified views – perhaps to look only at mean-sea-level pressure, for example – it’s also good sometimes to embrace the complexity, and remind ourselves why we need a supercomputer to keep track of it all.

So this time I’ve put as much as possible in the video: sea-ice, wind speed and direction, temperature and even rainfall. It’s still only a tiny fraction of the full three dimensional atmospheric state that our forecast models have to simulate, but there’s plenty to look at: We can see not only the small-scale complexity of the winds, but also some larger-scale patterns: the strong clockwise circulation around Antarctica formed by the southern hemisphere westerlies, the cyclones forming in that strong flow, and atmospheric waves folding outwards.

This isn’t really old weather, it’s almost new – from only last month. But I used this example because it illustrates that the weather is not only complicated and interesting, it also matters. If you set the video to September 16th you’ll see a low pressure (clockwise circulation) off Marie Byrd land, linking with a high pressure (anti-clockwise circulation) in the south-east Pacific. These combined to channel cold Antarctic air up toward central Chile, which contributed to a late frost which cost their fruit industry an estimated $1 billion. Expect to pay extra for peaches, cherries, and even Cabernet Sauvignon, as a result.

Brightening the world

We launched oldWeather three years ago today (October 12th, 2010). It was an exciting but scary moment – would she float? We’d done everything we could, but you’re never quite sure until the splash has settled.

One thing we did know at launch was where we were going: The map of past climate variability and change contains some very large blank areas – great expanses of space and time where we knew almost nothing of what the weather had done. Ours was a voyage of exploration: We would sail, via the archives, into these regions and rescue their weather observations, adding systematically and permanently to the scientific records on which our understanding of the climate is based.

And it’s worked very well. As with any research project we’ve encountered plenty of surprises along the way, but they’ve been good surprises – we knew about the weather in the logs, but we didn’t realise just how much else was in there. So we’ve added detailed ship histories, maps, geographical databases, illustrations of the course of WW1, tales of life on board, …

But our primary aim is still the weather, and we’ve recovered an enormous account of historical weather information, more than 1.6 million new observations from our original set of Royal Navy logs alone. These new basic observations are a permanent foundation on which scientists all over the world can build new reconstructions and products, and today we can see such a building appear.

Gil Compo and colleagues, from NOAA/CIRES/University of Colorado, are using our new observations in an atmospheric reanalysis (20CR). Essentially they combine surface weather observations (such as ours) with information on sea temperature and sea-ice, and a physical model of the atmosphere, to make a detailed and comprehensive picture of the global weather. It takes some of the world’s largest supercomputers to do this analysis: 20CR was produced at the US National Energy Research Scientific Computing Center and the US Oak Ridge Leadership Computing Facility. But it’s worth the effort – not only do they make a global weather reconstruction, but they also calculate the accuracy of their reconstruction, and we can compare their new reconstruction with one they made earlier, to see how much difference our observations have made.

So the video above has four components:

  1. The weather. The reanalysis calculates everything about the weather: winds, temperatures, clouds, rainfall, the jet stream, … but I can’t show all that in one video so we’re only seeing mean-sea-level-pressure. The solid black contours show where this is low (bad weather), and the dashed contours where it is high (good weather).
  2. The observations. Grey dots mark observations we’ve had since before oldWeather started. Yellow dots mark new observations. Most (but not quite all) new observations are from oldWeather. (We are only part of a wider recovery program).
  3. The fog of ignorance. Grey fog marks the areas where we still don’t have enough observations to say exactly what the weather was doing.
  4. The glow of discovery. Yellow highlighting marks the areas where the reconstruction is much better than it was before (mostly because of our new observations).

That’s a lot to get in one image, but it’s the yellow that matters. Our work has cleared the fog, and illuminated the weather, over a huge area of land and ocean. The improvement stretches over about 20% of the Earth’s surface – more than 100 million square kilometres – and is there for every hour of the 9+ years covered by the Royal Navy logs we read.

That’s not a bad return for our three years hard work.

Melt season

Two summers: On the left, 1980; on the right, 2012.
(The picture is of the Arctic Ocean (with Iceland at the bottom and Alaska towards the top). It is about 3000 miles from side to side).

We tend to use ‘global warming’ and ‘climate change’ almost as synonyms, but that’s not quite right: the climate is changing, and one of the ways we see that change is as an increase in global mean temperature. We like global temperature as a measure partly because it is relatively well observed and understood (thanks, in a small part, to our contributions), but climate change is also showing itself in other ways, some of them more dramatic.

Every year in the Arctic, the sea-ice starts to melt in March and continues to retreat through the summer, reaching its minimum extent in September. Since 1979 we’ve been able to watch the change by satellite, and even over the 30-odd years of satellite observations we’ve seen some big changes, particularly in the summer ice coverage:

This is one reason why we are now concentrating on polar data. Arctic sea-ice is harder than global temperature – to measure, to understand, and to predict. So more observations are particularly valuable. And because changes in ice cover can be so large, we can make useful comparisons to modern records even with a limited set of ship observations: in 2012 the Northwest passage was clear of ice – it’s certain that William Parry, John Franklin, Roald Amundsen, and even our own Thetis, met very different conditions.

oldWeather Arctic


Volunteers wanted for vicarious journeys. Bitter cold, long months of complete darkness, constant danger, safe return doubtful. Honour and recognition in case of success.

Today we launch a new fleet on the focus this time is on Arctic voyages, and the logbooks are from the collection of the US National Archives in Washington. We’ve ships from the Revenue and Coast Guard, the Navy, and the Coast Survey; and they include some famous names and some exciting voyages.

The Arctic is very sensitive to climate variability and change. This year (2012) was a record year for sea-ice: there was less sea-ice this September than for any other year for which we have good satellite records. But the satellite records only go back to 1979, and we need many more than 30 years of records to really understand how the Arctic climate behaves. This means we need to rescue the weather records of the people who travelled there – to read the logs of Arctic voyages.

If you joined in the original oldWeather, you’ll notice some differences in this new version: There are fewer ships (at least to start with, we’ll be adding more regularly), but the records for each ship usually cover many years, so we have just as many pages to read. These logbooks are also older (back to 1850 in some cases), and differently laid-out, so we’ve had to change the way you enter data: Basically it’s the same – select the location on the log page with an important record and then type the record into the pop-up box – but the details have changed. So whether you’re a new recruit or an old hand, please experiment until you get used to it – there is a tutorial to guide you, and help and encouragement on the project forum.

We’re still looking for all weather records, and anything else you read and think is interesting or notable. There will be plenty of notable historical events: the dangers of sailing through the ice add a lot of drama to the stories in the logs – whether you prefer the daring rescue by USRC Bear, ice and fire on the USS Rodgers, or the so-far-unknown adventures of less famous ships.

The voyage of USRC Thetis – April to September 1884

We’ve been running the beta-test of oldWeather Arctic for several weeks now, and we’ve accumulated plenty of completed log pages – that’s log pages that have been transcribed by the three people we need to get reliable results. So it’s time to have a good look at the results we’re getting: Is our new interface collecting the transcriptions properly, and are the transcriptions we’re getting accurate and useful?

This time I’ve tried to show explicitly the link between what we’re doing on the website and the numbers that are going to the science team. The image below shows this for a single log page from USRC Thetis (click on the image for a bigger version).

Log of USRC Thetis for 1884-05-05

The log page for USRC Thetis from May 5th 1884 – as transcribed through The coloured boxes overlain on the log page are those drawn by the transcribers (blue for weather, red for locations, and grey for dates). The text on the right hand side shows the information collected from the transcriptions.
(Bigger image, Video showing all pages.)

The left hand side of the image shows what we’re doing on – a log page marked-up with the locations of valuable data. (This time I’ve looked only at the dates, positions, and hourly weather observations – the historical events and informal weather records (including ice observations) are just as important but I didn’t have space for them.) As each page is transcribed by at least three people, there are usually three selections for each record. The right hand side shows the values extracted from the transcriptions.

For this page it’s working very well: we’re getting the detailed weather and ship-position information we need. Of course, that’s just one page – we need to do that for every page, and that’s shown in this video, which shows the transcribed data streaming out of the log consistently and accurately. All our hard work transcribing is delivering the detailed weather records the scientists need.

We can also look at the 2718 new weather observations we’ve rescued from the Thetis so far. How do they compare with more recent observations? Were the sailors on this ship careful and accurate observers? To judge this I like to compare the oldWeather observations (red points in the figure below) with modern records.

Observations made by USRC Thetis on her voyage from New York to Smith Sound in summer 1884 (red dots), compared with modern climatologies (black lines).

The top left image shows the route of the ship: from New York up through the Labrador Sea and Baffin Bay and back – a true Arctic voyage. The bottom right image shows that measured air temperatures were typically lower on the voyage in 1884 than the average (climatology) for the last few years – an intriguing result (though there are many possible reasons for it). Top right and bottom left are air pressure and wind speed, these are harder to compare because for pressure and wind we expect bigger differences between an observation (a point value) and a climatology (an average over several years). Rather than going into details I’ll just say that I’m very pleased with these results too; this comparison is exactly what we’d expect from good-quality, useful observations.

So well done USRC Thetis and all who sail with her – both her original crew who took the observations, and the oldweather crew led by Lekiam, Jelliott8 and lollia paolina.

The transit of Venus

The weather in Exeter yesterday was best described as “ocr“, so I missed the transit this time. Fortunately, the skies were clear back in 2004, and I remember the experience of peering through a pair of binoculars equipped with a sun-filter and seeing the small black dot of Venus silhouetted against the sun.

The transit of Venus is a periodic event, and the big year was in 1769. I understand that the astronomers valued the transit as a way to get a handle on the size of the universe; but the real virtue was that it provided an excuse for the British Government to send an expedition down into the South Pacific. That expedition was commanded by James Cook, and it started the career of the greatest explorer of them all.

Of course, as a naval officer (Lieutenant, at the time), Cook kept a journal. If your eyesight is up to it, you can read his account of June 3rd 1769 in the original handwriting; but I admit that I looked up Project Gutenberg’s transcription:

Saturday, 3rd. This day proved as favourable to our purpose as we could wish. Not a Cloud was to be seen the whole day, and the Air was perfectly Clear, so that we had every advantage we could desire in observing the whole of the Passage of the planet Venus over the Sun’s Disk. We very distinctly saw an Atmosphere or Dusky shade round the body of the planet, which very much disturbed the times of the Contact, particularly the two internal ones. Dr. Solander observed as well as Mr. Green and myself, and we differ’d from one another in Observing the times of the Contact much more than could be expected. Mr. Green’s Telescope and mine where of the same Magnifying power, but that of the Doctor was greater than ours. It was nearly calm the whole day, and the Thermometer Exposed to the Sun about the Middle of the day rose to a degree of heat we have not before met with.

The ideal weather observer does not expose his thermometer to the sun (shade temperatures please), so perhaps it’s no great loss that Cook’s journal does not contain regular weather observations. For those, we must turn to the Master’s log of HM Bark Endeavour (the closest equivalent to the familiar modern-day logs). For the day of the transit, this records “Little wind and variable with fine pleasant clear weather”. I reckon that’s “Lt. Airs, Var., 1, b” in our notation. Sadly, none of the logs contain regular thermometer or barometer observations – Cook did better on his subsequent voyages – but we do get wind speed and direction reports for every day.

Analysis of the oldWeather data

Now that we’ve completed the original batch of logs we were working on in oldWeather, we’ve started to release the new weather observations recovered for use in scientific investigations. Leading the way in such investigations is Ed Hawkins of the University of Reading, who’s started a series of posts on his blog describing the value of oldWeather for the study of Arctic climate and sea-ice, the North Atlantic Oscillation, and reducing uncertainties in Atlantic pressure fields.