Archive by Author | Philip

Collectively awesome

We launched the new a month ago, which means that the volunteers using the site have provided quite a bit of new data, and we can start to analyse it. This is one of my favourite moments in any project – first blood, when we get the initial sense of what we’ve got, how it’s going to work, what we can learn from it.

One of the golden rules of statistical analysis is “first plot the data” – always start by making a simple visualisation, so you can be sure you understand what you’ve got, and you’re not missing anything obvious. But the oldWeather data is not easy to plot: the database contains records from hundreds of people making thousands of annotations on dozens of different logbook pages; what, exactly, should we look at?

So I’ve taken inspiration from Listen to Wikipedia, and asked ‘what would it look like if we could see (and hear) the data as it came in – in (accelerated) real time?’ The video below shows every contribution to over a three hour period on December 3rd 2015. The number of pages shown is the number of volunteers contributing at each point in time. Each box drawn, and sound played, is one annotation, a contribution to the project. Blue boxes contain weather data, yellow boxes ship positions, orange boxes dates, and red boxes other events; pages that have moved on to the transcription phase have grey boxes.

Listen to oldWeather: December 3rd 2015.

December 3rd was when we launched the new site, so we can see a large change in the number of people participating as they learn about the launch. It’s instantly clear that it’s working – we are collecting annotations and transcriptions in quantity, as we hoped. There is much to be learned from careful examination of visualisations like this, but mostly I think it shows the power of the project – the awesome capability of collective public science.

Note that this shows only data from the new Panoptes version of oldweather. It does not include data from the whaling site – I’m still working on that.

A double century of centennial records

Australian cricketer Herbie Collins

Very few achieve 200

What was life really like in the Royal Navy 100 years ago? Where did the ships go? How did the crews spend their days? What were the noteworthy, and the routine, events in their lives?

The Royal Navy logbooks we worked through in the original version of oldWeather provide a uniquely powerful insight into these questions – they are primary records of exactly what happened. But they are not easy to use – hard to read, not indexed, or searchable, and often full of obscure technical language.

When we transcribed the weather in the logs we caught many of the historical events as well, and we were able to make a formatted history file for each ship – linking each logbook page image to transcribed events and information from that day, and we assembled those ship histories on our partner website

Those history files made from the raw transcriptions are a good start, but they are far from perfect: Some events we caught cleanly, some only half-stopped, and our decisions on what to leave were usually good, but not always. So our team of volunteer editors have been working through the raw files editing and improving them: reviewing the decisions made in the heat of transcription, correcting mistakes, merging multiple versions, adding missing events, incorporating pithy commentary and expert summaries of key points, and adding maps of the ship journeys.

As so often with oldWeather, this has been a lot of work – a major task tackled with care and patience by an increasingly-expert team of volunteers. Their achievement is clear to see, comparing the edited histories (in bold on this page) with the raw versions shows a huge improvement in clarity, accuracy, completeness, and value. And the score of the editing team has mounted steadily – they have just released their 200th edited ship history.

To get to 200 ships edited is an awesome achievement, but of course we still have power to add: HMS Cricket is done, as are Cardiff, India, New Zealand and Sydney; but Dunedin, Durban, Perth, Delhi and Capetown are yet to be conquered. Are you available for selection?

There she blows!

We’ve spotted the first signs of new ships on oldWeather. We are partnering with the New Bedford Whaling Museum to read and transcribe their collection of whaling ship logs. Whalers are not just historically important and interesting, their hunting grounds were well into the Arctic, so we’re hoping to get some particularly good sea-ice records from these logs.

These new logs look quite different to the government ship logs we are used to – they don’t have a uniform page structure with tables of data – so we are adding them in their own separate section, and the way to mark the page and enter data is a bit different. Please have a look at the new logs, and experiment with the new interface – it will take a while to get used to, but help and advice is available, as always, through the forum.

We are also adding one other new thing: a way to add our own value to individual log images. We’ve always concentrated on transcribing the contents of the logs, but up to now we have not had a good way to add our own comments to them: or mark interesting pages, or flag mistakes made by the log-keeper, or tag pages to make groups. We are adding the ‘Talk’ system (as featured on other Zooniverse projects) to oldWeather to let us do this, and I’m really looking forward to what we can do with it.

If you like the old oldWeather, don’t worry, it’s still there as before, but now we have the added opportunity of oldWeather-whaling.

Floating Forests

Rockfish around kelp Monterey Bay Aquarium
Kelp is, perhaps, more important than you might guess: Not only does it thicken your toothpaste, it supports whole marine ecosystems where it grows. It is important enough to have a Zooniverse project devoted to watching it grow, through Landsat images.

Satellite imagery is a great way to monitor the world – providing frequent, comprehensive pictures of the whole planet. But in-situ observations also have their place: people on the ground, interacting directly with the system being monitored, can often provide a detail and precision that the satellite records lack.

One of the unexpected joys of oldWeather is that it provides in-situ observations of a vast range of different things. Most often kelp is mentioned in the logs simply as a highlight of a day at sea:

Fine weather. Light breeze from South. At 2.30 took in and furled the sails. Passed a piece of kelp. [Yorktown, May 1892.]

4 to 8 a.m. Overcast but pleasant. Airs from NE. Passed some kelp. [Same, a day later.]

Sometimes we see the kelp interacting with the environment:

Saw a large patch of kelp with a dozen seals hauled out on it. [Rush, June 1891]
Sighted a whale and a bunch of kelp. [Yorktown, May 1892]
At daylight passed much drift kelp, to one large batch a boulder about 3 ft in diameter was attached [Patterson February 1885]

Occasionally they do seem to be actively surveying it:

Steaming along at various speeds, locating outer limit of kelp beds off La Jolla, fog gradually increasing, log hauled in. [Pioneer, spring 1923].
Continued sounding passing inside of Aleks Rock. No signs of kelp were seen. [Patterson].

But the most interesting mentions feature it as a hazard to navigation. I suspect most of our log-keepers would see definite benefits in any decline of kelp:

Slowed down a few minutes on account of kelp. [Concord, August 1901]
4:15 Kelp ahead, full speed astern … Ran about 1/2 mile SWxW and ran into kelp again. Wreck bore E 1/2 N. Stopped and backed away from it [Patterson].
found four masted schooner “Watson A. West” in the kelp on the outer edge of the shoal, broadside to the beach, close in and in dangerous position [Unalga, October 1916].
Between six and seven o’clock, patent log registered only 3.9 knots: hauled in rotator and found it fouled with kelp; cleared it, and allowed 2.6 knots for the discrepancy. [Commodore Perry, July 1896].
Found spar buoy #16, two hundred yards NE of true position and in kelp. [Commodore Perry, February 1903].
At 10.36 sighted what appeared to be a pinnacle rock. Stopped ship lowered boat and after inspection the object proved to be a much worn spar, heel up, with kelp attached. [Yorktown, June 1894].

We don’t have that many observations of kelp – we probably won’t be much help to the Floating Forests team mapping the distribution, but we do have our own viewpoints to add – aspects that the satellites will never see:

Thanks to all those who’ve transcribed such records from the logs.

Mentioned in despatches

We're in there: Figure 9 from Cram et al. 2015.

We’re in there: Figure 9 from Cram et al. 2015.

oldWeather does not produce that many research papers: The process of science is changing fast at the moment – moving away from individual researchers and small groups working independently, and towards much larger consortia working together with big datatsets on major problems – fewer small papers, more big science. As a major data provider, closely linked into the international climate research community, we fit nicely into this new model.

But the academic papers industry is still out there, and now and again one appears that involves us directly. One of the major datasets we contribute to is the International Surface Pressure Databank, and Tom Cram and colleagues have a new paper about the data (including our contributions) and how it’s assembled and distributed through the excellent Research Data Archive at NCAR. It’s open access – free for all to read (here’s the link) – why not check out the oldWeather references and see exactly how our results are being used by researchers.

International Archives Day

Today, 9th June, is International Archives Day, where archives around the world unite to celebrate their remarkable resources by sharing an iconic image from their collection. oldWeather is no archive, but we certainly do appreciate them, and we aim to add value to their collections through our transcription work.

So here’s an image in which we saw some special value – this is from the collection of the UK National Archives: The logbook of HMS Tarantula, from August 1920. Thanks to generations of archivists for preserving these records for use today; without their skill and dedication this record, and all the others from which we have learned so much, would have been lost forever.

Logbook of HMS Tarantula, August 1920

On Sunday 29th Aug. aneroid was compared with Standard Mercurial in HMS Carlisle and found to be reading 0.26 ins. too low. An examination of the aneroid showed it to contain an ants’ nest and be otherwise defective. This aneroid is being surveyed and until a new one is received the Captain’s private Baragraph will be used. This barograph was corrected on 29th August.

DIY for digital humanists

oldWeather is, of course, a science project. Or at least mostly science. Or … OK, we do lots of things, plenty of science, but also work on history, archives, data visualisation, information retrieval, human-computer interaction, motivation, and much else. In fact, we are something of a poster child for the young research field of digital humanities.

If you are a researcher interested in the digital humanities, you might want to come to this year’s Digital Humanities at Oxford Summer School. In particular, Victoria Van Hyning, one of our friends from the Zooniverse is running a workshop on Crowdsourcing for Academic, Library and Museum Environments where participants will learn how to build their own citizen science projects, using the same Zooniverse technology that’s being used for developing oldWeather.

oldWeather has proved both fun and productive for almost everyone involved (very hard work, too, of course), so I’m keen to encourage other such projects. Try it – there must be many fascinating object collections that would benefit from examination by several hundred eyes.

All will be assimilated

Weather from a scout run of the 20th Century reanalysis, using the US oldWeather observations.

The observations we are recovering will have many uses, but one is clear – they will be assimilated into collective reconstructions of historical weather (reanalyses). We’ve done this already with the original set of Royal Navy ships, and now we are starting to use the US Arctic ship observations in the same way. Leading the way, again, is the 20th Century reanalysis (we are working on nineteenth century records at the moment, but it’s too late to change the name now); the video above shows a new reconstruction of the weather for 1880 and 1881, and stars three of our ships (red dots) Jeannette, Corwin, and Rodgers. As well as the wind (vectors), temperature (colours), sea-ice, rain (black shading), and observations used (dots), it shows grey fog masking the areas where the reconstruction is still very uncertain (because there are no observations there – yet).

The improvement from our new observations can be seen clearly in the gaps in the fog surrounding each red dot, but the effect can be seen even more starkly if we follow one ship:

Observations from the arctic voyage of the USS Jeannette.

The map shows the route of the ship and the sea-ice for the period. The graph shows sea-level pressure, and near-surface air temperature, as observed by the ship (red dots), as reconstructed by 20CR version 2c (not using the ship’s observations: light grey band), and as reconstructed by a scout version of 20CR assimilating the Jeannette’s pressure observations (dark grey band – starts in Nov. 1879). Small yellow dots mark the other observations available to the reanalyses. The narrowing of the band indicates a reduction in our uncertainty about the weather – an increase in the value of the reconstruction.

So far we’ve only captured those three ships, but we’ll get to the rest – their observations and historical distinctiveness will be added to our reconstructions. Resistance is futile.

1085 conversations

Gil, Gina, & Mark by Kevin's banner

And straight against that great array Forth went the dauntless Three.
Gil (left), Gina, and Mark.

Our representatives at last weekend’s Smithsonian Arctic Spring Festival had a busy and succesful time: On Friday they welcomed 375 visitors, and on Saturday 710; including people not only from around the U.S, but also from Australia, India, Poland, Russia, and Korea. 710 people over a daily stint (10:00 am to 4:00 pm) averages at two people every minute for the whole six hours. Even though the visitors mostly came in groups and couples, that’s still a lot of talking – it’s hard work, even for a fun event.

But of course we have plenty of enthusiasm and a lot to talk about, and with Mark, Gina and Gil all present we had experts covering the whole sweep of the project, from the original records at NARA, through the logbook imaging and organisation, to the climate reanalysis outputs.

Gil talks to some colourful and enthusiastic visitors

Gil talks to some colourful and enthusiastic visitors

oldWeather at the Smithsonian spring festival

Old Weather: Our Weather's Past, the Climate's Future

This weekend – Friday 8th to Sunday 10th May – the Smithsonian National Museum of Natural History, in Washington DC (USA), is holding an Arctic Spring Festival (all welcome, free entry).

There’ll be lots going on: music, crafts, art, dance performances, …, and also an opportunity for visitors to meet and talk to scientists researching the Arctic. oldWeather will be there: Kevin has organised our display, with the video above and some good posters – and Gina, Mark, and Gil will be there in person, ready to talk about all aspects of the project.

So please do drop in, or tell your friends – we’ll be happy to see you.