oldWeather Arctic

Volunteers wanted for vicarious journeys. Bitter cold, long months of complete darkness, constant danger, safe return doubtful. Honour and recognition in case of success.

Today we launch a new fleet on oldWeather.org: the focus this time is on Arctic voyages, and the logbooks are from the collection of the US National Archives in Washington. We’ve ships from the Revenue and Coast Guard, the Navy, and the Coast Survey; and they include some famous names and some exciting voyages.

The Arctic is very sensitive to climate variability and change. This year (2012) was a record year for sea-ice: there was less sea-ice this September than for any other year for which we have good satellite records. But the satellite records only go back to 1979, and we need many more than 30 years of records to really understand how the Arctic climate behaves. This means we need to rescue the weather records of the people who travelled there – to read the logs of Arctic voyages.

If you joined in the original oldWeather, you’ll notice some differences in this new version: There are fewer ships (at least to start with, we’ll be adding more regularly), but the records for each ship usually cover many years, so we have just as many pages to read. These logbooks are also older (back to 1850 in some cases), and differently laid-out, so we’ve had to change the way you enter data: Basically it’s the same – select the location on the log page with an important record and then type the record into the pop-up box – but the details have changed. So whether you’re a new recruit or an old hand, please experiment until you get used to it – there is a tutorial to guide you, and help and encouragement on the project forum.

We’re still looking for all weather records, and anything else you read and think is interesting or notable. There will be plenty of notable historical events: the dangers of sailing through the ice add a lot of drama to the stories in the logs – whether you prefer the daring rescue by USRC Bear, ice and fire on the USS Rodgers, or the so-far-unknown adventures of less famous ships.

The voyage of USRC Thetis – April to September 1884

We’ve been running the beta-test of oldWeather Arctic for several weeks now, and we’ve accumulated plenty of completed log pages – that’s log pages that have been transcribed by the three people we need to get reliable results. So it’s time to have a good look at the results we’re getting: Is our new interface collecting the transcriptions properly, and are the transcriptions we’re getting accurate and useful?

This time I’ve tried to show explicitly the link between what we’re doing on the website and the numbers that are going to the science team. The image below shows this for a single log page from USRC Thetis (click on the image for a bigger version).

The log page for USRC Thetis from May 5th 1884 – as transcribed through oldweather.org. The coloured boxes overlain on the log page are those drawn by the transcribers (blue for weather, red for locations, and grey for dates). The text on the right hand side shows the information collected from the transcriptions.
(Bigger image, Video showing all pages.)

The left hand side of the image shows what we’re doing on oldweather.org – a log page marked-up with the locations of valuable data. (This time I’ve looked only at the dates, positions, and hourly weather observations – the historical events and informal weather records (including ice observations) are just as important but I didn’t have space for them.) As each page is transcribed by at least three people, there are usually three selections for each record. The right hand side shows the values extracted from the transcriptions.

For this page it’s working very well: we’re getting the detailed weather and ship-position information we need. Of course, that’s just one page – we need to do that for every page, and that’s shown in this video, which shows the transcribed data streaming out of the log consistently and accurately. All our hard work transcribing is delivering the detailed weather records the scientists need.

We can also look at the 2718 new weather observations we’ve rescued from the Thetis so far. How do they compare with more recent observations? Were the sailors on this ship careful and accurate observers? To judge this I like to compare the oldWeather observations (red points in the figure below) with modern records.

Observations made by USRC Thetis on her voyage from New York to Smith Sound in summer 1884 (red dots), compared with modern climatologies (black lines).

The top left image shows the route of the ship: from New York up through the Labrador Sea and Baffin Bay and back – a true Arctic voyage. The bottom right image shows that measured air temperatures were typically lower on the voyage in 1884 than the average (climatology) for the last few years – an intriguing result (though there are many possible reasons for it). Top right and bottom left are air pressure and wind speed, these are harder to compare because for pressure and wind we expect bigger differences between an observation (a point value) and a climatology (an average over several years). Rather than going into details I’ll just say that I’m very pleased with these results too; this comparison is exactly what we’d expect from good-quality, useful observations.

So well done USRC Thetis and all who sail with her – both her original crew who took the observations, and the oldweather crew led by Lekiam, Jelliott8 and lollia paolina.

One million, six hundred thousand new observations

For all its charm, oldWeather is a quantitative science project, so we shouldn’t be surprised that so many of the blog posts feature numbers: We’ve had 7s versus 4s, 3 transcriptions not 5, our first 8000 transcribers, 476 sick sailors, the first 120,000 observations, another 3000 logs, 63 voyages, 97% accurate, 1.85 characters, 127 games of cricket, 0.02953 inches, 150 ships, and 1 million pages.

My excuse for this nostalgia is that we’ve finished! We’ve done all our logs – every page, of the thousands of Royal Navy logs we’ve been looking at, has been read and transcribed by at least three people. And in the process we’ve generated one more impressive number – something over 1 million, six hundred thousand new weather observations (1,659,212 as I write this, but I’m always a bit behind with the analysis, so it’ll be a few more when we’ve got them all). Looking right back to the beginning, and the metaphor of the fog of ignorance, 1.6 million new observations is a lot of huffing and puffing, and many square miles of fog will be cleared by our efforts. Completing now is very timely, because the next generation climate reanalysis is starting this August, so we’re just in time to feed our results into a whole new sequence of climate research.

Every science project aims to change the world, and oldWeather has succeeded in this more profoundly than anyone expected: we hoped to recover weather records which were badly needed by climate researchers, and perhaps to get a bit of useful historical information along the way. We’ve got the weather records, and richer and more detailed and extensive historical infomation than we expected, but we’ve got much more than that: we’ve grown an amazing community of project participants, and we’ve planted the seeds for several other projects with similar aims.

Thousands of people have participated in the project, and many people have comitted lots of time and effort to it – bringing in their own expertise and combining it with new knowledge learned from the logs. Sharing that knowledge on the project forum has built a community expertise in weather, history, geography, marine and naval details, …, which is finding much more value in the logbooks than we initially expected. It’s not all work either – we’ve made art, and poetry, with a good deal of fun along the way.

And I’m not the only one who’s been watching the project with strong admiration – researchers around the world have looked at oldWeather, and then turned back to their own archival records thinking ‘if oldWeather can do it, why not…’. Most of the resulting projects are still embryonic and we can’t yet say which will grow to full life, but don’t be surprised in future to see projects popping up based on Dutch logbooks, French logbooks, and a wide variety of non-meteorological records. (There is at least one follow-on project that will definitely appear – our own: oldweather.org the next generation is already available – but that’s material for another post).

Thanks and congratulations from the science team, the climate science community, the marine historical community, and many newly-inspired citizen scientists. Three cheers for oldWeather, there’s none like us.

See also: Observations by ship, the fog of ignorance, and the credits reel.

The transit of Venus

The weather in Exeter yesterday was best described as “ocr“, so I missed the transit this time. Fortunately, the skies were clear back in 2004, and I remember the experience of peering through a pair of binoculars equipped with a sun-filter and seeing the small black dot of Venus silhouetted against the sun.

The transit of Venus is a periodic event, and the big year was in 1769. I understand that the astronomers valued the transit as a way to get a handle on the size of the universe; but the real virtue was that it provided an excuse for the British Government to send an expedition down into the South Pacific. That expedition was commanded by James Cook, and it started the career of the greatest explorer of them all.

Of course, as a naval officer (Lieutenant, at the time), Cook kept a journal. If your eyesight is up to it, you can read his account of June 3rd 1769 in the original handwriting; but I admit that I looked up Project Gutenberg’s transcription:

Saturday, 3rd. This day proved as favourable to our purpose as we could wish. Not a Cloud was to be seen the whole day, and the Air was perfectly Clear, so that we had every advantage we could desire in observing the whole of the Passage of the planet Venus over the Sun’s Disk. We very distinctly saw an Atmosphere or Dusky shade round the body of the planet, which very much disturbed the times of the Contact, particularly the two internal ones. Dr. Solander observed as well as Mr. Green and myself, and we differ’d from one another in Observing the times of the Contact much more than could be expected. Mr. Green’s Telescope and mine where of the same Magnifying power, but that of the Doctor was greater than ours. It was nearly calm the whole day, and the Thermometer Exposed to the Sun about the Middle of the day rose to a degree of heat we have not before met with.

The ideal weather observer does not expose his thermometer to the sun (shade temperatures please), so perhaps it’s no great loss that Cook’s journal does not contain regular weather observations. For those, we must turn to the Master’s log of HM Bark Endeavour (the closest equivalent to the familiar modern-day logs). For the day of the transit, this records “Little wind and variable with fine pleasant clear weather”. I reckon that’s “Lt. Airs, Var., 1, b” in our notation. Sadly, none of the logs contain regular thermometer or barometer observations – Cook did better on his subsequent voyages – but we do get wind speed and direction reports for every day.

1 million pages

The 15th edition of the Encyclopaedia Britannica runs to 32 large volumes, each of about 1000 pages. The million pages of logs we’ve transcribed would take up more than 30 times as much space on the shelf.

Today oldWeather has passed another remarkable milestone: we’ve now transcribed 1 million logbook pages. 1,000,000 or 10⁶ – however you write it, that’s a big number.

The logbooks have large pages, so think of a big, heavy book – let’s say a volume of the old Encyclopædia Britannica. Those have about 1000 pages each, so we’d need about 1000 such volumes to make up a million pages – more than 30 copies of the entire Encyclopaedia (15th edition).

Alternatively, consider the average American, who reads 9 books a year. If a typical book is 300 pages in length, we’ve done as much reading as that average American does in about 350 years. We haven’t been skimming the logs, either: It takes, on average, about 2 minutes to read and transcribe each log page.

So we’ve spent 2 million minutes with our collective nose in a log – a task which would have been quite impossible without the combined efforts of thousands of project participants. And what treasures we’ve found in there: As well as millions of invaluable weather observations, we’ve followed stories of war, sickness, celebration, drunkenness, heroism, tragedy, partying, … Surely a better read than any novel.

The image at the head of this post was shamelessly stolen adapted from Wikipedia.

An excellent series of new logbooks

You’ve probably noticed that we’ve published another instalment in the ongoing series of Royal Navy logbooks on oldWeather. For those keeping count, this is batch four of oldWeather phase two, and the unifying theme of this group of books is that the ships travel far from home indeed – these are voyages that include operations in the Pacific.
There’s the usual range of ships to choose from, from the Battlecruisers Renown and Repulse, to the Torpedo Boats and Gunboats doing inshore work in China; all of them helping to map the climate of an ocean that was named for its good weather, but does not always live up to its name.
We expect these logbooks to describe operations in the Pacific, but of course we won’t really know what they are up to (and where) until we’ve read them. So expect a Surprise – even if you don’t earn a Post as Captain, maybe you’ll read of a voyage past Mauritius, or Desolation Island, or encounter some strange Fortune of War on the way to the Far Side of the World.

Analysis of the oldWeather data

Now that we’ve completed the original batch of logs we were working on in oldWeather, we’ve started to release the new weather observations recovered for use in scientific investigations. Leading the way in such investigations is Ed Hawkins of the University of Reading, who’s started a series of posts on his blog describing the value of oldWeather for the study of Arctic climate and sea-ice, the North Atlantic Oscillation, and reducing uncertainties in Atlantic pressure fields.

New Mediterranean Voyages

HMS Ben-my-Chree, one of our new ships.

For the last few weeks we’ve been watching the ‘% completed’ figure on the Old Weather front page creep ever closer to 100. We’re very close to completing the task we took on on October 12th, 2010, by transcribing every single log page (750,000 transcriptions). That’s an awesome achievement – take a moment to feel the victory.

But we’re not ready to decommission the project – there’s more yet to learn about the climate and the history of the early twentieth century. There are many logs in the National Archives that we’ve never looked at: some from our old friends, and some from new ships we’ve never before followed. So with support from JISC, we returned to the National Archives, camera in hand, to collect a new set of logbook images – and we’re launching the first batch of them today as the start of oldWeather phase II. (There will be more to come).

This first batch of new logs is timely for those of us in the North, feeling the cold of winter set in, because these are voyages in the Mediterranean. Forget the chill and the dark, and travel vicariously through warm and history-filled seas. If you have fond memories of a previous voyage on HMS Africa, Britannia, Caesar, Liverpool, or Theseus; furlough is over, your ship needs you – please report to your former station at your earliest convenience. If you are looking for new company and new adventures, HMS Aphis, Ark Royal, Barham, Ben-my-Chree, Blenheim, Canterbury, Castor, Galatea, Grafton, Minto, St George, and Warspite launch today for the first time with us. The point of departure, in all cases, is http://oldweather.org.
.

One year of oldWeather

We started our voyage one year ago (12th October 2010), and we’ve come a long way since then. According to the standard baby development milestones, we should now be able to explore new objects by poking with one finger, crawl up and down stairs, and reach out and grab things we want.

That article seems to have left out the standard ages for ‘transcribed 685,000 log pages’, ‘generated one million new historical weather observations’, and ‘built a thriving community of 10,000 historical climatologists’. But I think it’s safe to say that not many one-year-olds have achieved as much – we are remarkable indeed.

Happy birthday oldWeather, it’s been a treat to grow up with you. And in common with most people of our age, I look forward to exciting new development in the near future.

oldWeather Credits Reel

One of the main uses of the weather observations that we are collecting is in new reanalyses – reconstructions of weather and climate over the last few decades or centuries. This week, dozens of scientists working on weather and climate reconstruction are meeting for a workshop on reanalyses and historical weather observations, hosted by the Royal Netherlands Meteorological Institute at De Bildt.

This is an opportunity to tell everybody working in the field just how much we’ve achieved with oldWeather over the last 11 months, so I’m giving a presentation highlighting our results. As you’ll have seen from earlier blog entries, there’s plenty to present – so my biggest challenge is in working out how to give credit to all the project participants: It’s a firm rule in science that you should credit all your collaborators in any project, but there are 9566 people who’ve made a significant contribution to oldWeather (at the last count). So to list them all I’ve borrowed a technique from the movies, and made a credits video – this video is being premiered at the meeting (as part of my talk).

Of course it’s not enough just to have lots of people involved, we’ve also got to generate lots of new scientific results. So I’ll also be showing another video – less detailed, but faster and much more colourful – showing the 841848 new weather observations that we’ve generated.