Archive | Data RSS for this section

Frost in the South

We’ve looked at the world from the top; this is the view from beneath: Antarctica in the centre, South America at top, South Africa right, Australia and New Zealand bottom left. Streamlines show near-surface wind, colours indicate temperature, dots mark rain and snow. All data are from the Met Office global analysis.

One reason why weather forecasting and climate research are hard is that the atmosphere is complicated: There’s a lot going on – all sorts of different motions and changes occurring simultaneously all over the world. So while it’s often useful to use simplified views – perhaps to look only at mean-sea-level pressure, for example – it’s also good sometimes to embrace the complexity, and remind ourselves why we need a supercomputer to keep track of it all.

So this time I’ve put as much as possible in the video: sea-ice, wind speed and direction, temperature and even rainfall. It’s still only a tiny fraction of the full three dimensional atmospheric state that our forecast models have to simulate, but there’s plenty to look at: We can see not only the small-scale complexity of the winds, but also some larger-scale patterns: the strong clockwise circulation around Antarctica formed by the southern hemisphere westerlies, the cyclones forming in that strong flow, and atmospheric waves folding outwards.

This isn’t really old weather, it’s almost new – from only last month. But I used this example because it illustrates that the weather is not only complicated and interesting, it also matters. If you set the video to September 16th you’ll see a low pressure (clockwise circulation) off Marie Byrd land, linking with a high pressure (anti-clockwise circulation) in the south-east Pacific. These combined to channel cold Antarctic air up toward central Chile, which contributed to a late frost which cost their fruit industry an estimated $1 billion. Expect to pay extra for peaches, cherries, and even Cabernet Sauvignon, as a result.

Brightening the world

We launched oldWeather three years ago today (October 12th, 2010). It was an exciting but scary moment – would she float? We’d done everything we could, but you’re never quite sure until the splash has settled.

One thing we did know at launch was where we were going: The map of past climate variability and change contains some very large blank areas – great expanses of space and time where we knew almost nothing of what the weather had done. Ours was a voyage of exploration: We would sail, via the archives, into these regions and rescue their weather observations, adding systematically and permanently to the scientific records on which our understanding of the climate is based.

And it’s worked very well. As with any research project we’ve encountered plenty of surprises along the way, but they’ve been good surprises – we knew about the weather in the logs, but we didn’t realise just how much else was in there. So we’ve added detailed ship histories, maps, geographical databases, illustrations of the course of WW1, tales of life on board, …

But our primary aim is still the weather, and we’ve recovered an enormous account of historical weather information, more than 1.6 million new observations from our original set of Royal Navy logs alone. These new basic observations are a permanent foundation on which scientists all over the world can build new reconstructions and products, and today we can see such a building appear.

Gil Compo and colleagues, from NOAA/CIRES/University of Colorado, are using our new observations in an atmospheric reanalysis (20CR). Essentially they combine surface weather observations (such as ours) with information on sea temperature and sea-ice, and a physical model of the atmosphere, to make a detailed and comprehensive picture of the global weather. It takes some of the world’s largest supercomputers to do this analysis: 20CR was produced at the US National Energy Research Scientific Computing Center and the US Oak Ridge Leadership Computing Facility. But it’s worth the effort – not only do they make a global weather reconstruction, but they also calculate the accuracy of their reconstruction, and we can compare their new reconstruction with one they made earlier, to see how much difference our observations have made.

So the video above has four components:

  1. The weather. The reanalysis calculates everything about the weather: winds, temperatures, clouds, rainfall, the jet stream, … but I can’t show all that in one video so we’re only seeing mean-sea-level-pressure. The solid black contours show where this is low (bad weather), and the dashed contours where it is high (good weather).
  2. The observations. Grey dots mark observations we’ve had since before oldWeather started. Yellow dots mark new observations. Most (but not quite all) new observations are from oldWeather. (We are only part of a wider recovery program).
  3. The fog of ignorance. Grey fog marks the areas where we still don’t have enough observations to say exactly what the weather was doing.
  4. The glow of discovery. Yellow highlighting marks the areas where the reconstruction is much better than it was before (mostly because of our new observations).

That’s a lot to get in one image, but it’s the yellow that matters. Our work has cleared the fog, and illuminated the weather, over a huge area of land and ocean. The improvement stretches over about 20% of the Earth’s surface – more than 100 million square kilometres – and is there for every hour of the 9+ years covered by the Royal Navy logs we read.

That’s not a bad return for our three years hard work.

Journey Plotter, or an Old Weather addiction story

[This post is from Maikel, who has come up with a new way of using and viewing the information we are collecting].

The voyage of HMS Birmingham (showing a few position errors) as seen in Journey Plotter.

The voyage of HMS Birmingham (showing a few position errors) as seen in Journey Plotter.

Having been active as an Old Weather transcriber and in editing the transcribed logs for display on Naval-History.Net, I started to be curious about the journeys of the Royal Naval ships.

Giving in to this form of Old Weather addiction, I started to create an application that could retrieve the position information for the vessels I had edited. Seeing the dry numbers being transformed into a 2-dimensional voyage on a map was such a satisfying experience, I just had to share it with others.

This resulted in Journey Plotter, a Windows application for plotting the journeys, or parts thereof, of Royal Navy ships from the World War 1 era. Journey Plotter makes use of data from original Royal Naval log-books that have been transcribed and edited by oldWeather volunteers, and then made available by Naval-History.Net. Journey Plotter also turned into a valuable tool for the log editors: Position mistakes of a vessel are easily overlooked if it’s just a number. Seeing a strange jump in a voyage makes it much easier to spot.

If you are interested in Journey Plotter, visit http://journeyplotter.ihostfull.com to learn more about it. I trust you will enjoy looking at the journeys of the Royal Navy vessels and/or have a useful tool during the editing of their voyages.

Maikel

Images from Journey Plotter are now being used in some of the ship history files we’re producing: E.g. HMS Birmingham, HMS Una.

Over on the left

All the ports and named places located by the project.

All the ports and named places located by the project.

Working on oldWeather is a pretty wide-ranging education: we’ve got weather and climate, history, naval and maritime operations – and, it turns out, a sizeable chunk of geography as well. The logs are thorough in recording the ship locations, but they sometimes think that it’s enough to mention that they’ve sighted Qeqertarsuup Tunua, or anchored off Changqingshaxiang. To follow along, and to map the ships and use their observations, we have to turn those mentions into latitudes and longitudes.

This is another area where human ingenuity and effort are vital, and it’s another job we are doing well: The section on the forum dedicated to geographical help has yielded another project output – a database of port and place locations which is valuable not just to us, but also to other researchers trying to find out where their ships are.

O corpora o mores

Selected n-grams from the Royal Navy logbooks.

Selected n-grams from the Royal Navy logbooks. (Via Wordle).


One of the fun and useful tools Google have provided for us is an n-gram viewer. N-gram is a ugly but short term meaning a phrase containing n words: so ‘rain’ is a 1-gram, ‘clear sky’ a 2-gram, and ‘overcast with squalls, hail, thunder and lightning’ a 7-gram. Google’s tool shows us how common selected phrases are, and how their use has changed over time – I’ve found intriguing results with meteorological instruments, and ship details, for example.

How common is the word ‘barometer’? begs the question ‘how common in what?’ – English newspapers?, Canadian novels?, Spanish poetry?. The block of words used for the search is called a corpus, and Google offers several to choose from; but we don’t have to use any of them, because we have made our own. We actually have two corpuses corpora to choose from: one from the Royal Navy WW1 logs, and a second from the US Arctic logs we are working on now. We don’t have a convenient web tool, but we can still search out the common words and phrases.

In the RN logs the most common 1-gram (word) is ‘to’, but if we disqualify ‘to’, ‘in’, ‘and’, ‘but’ and the like, the top 10 are ‘sick’, ‘ship’, ‘HMS’, ‘list’, ‘proceeded’, ‘hands’, ‘discharged’, ‘joined’, ‘arrived’ and ‘left’ – almost enough in themselves to give a sense of the naval language. Looking at popular longer phrases gives an even better picture: ‘joined ship’, ‘weighed and proceeded’, ‘hands employed cleaning ship’, ‘came to with port anchor’; all the way up to the likes of ‘Ship in dockyard hands for refit. Ship’s company employed as required on board and accommodated in sailors home’.

The US Arctic logs are much more variable, and we have not yet accumulated a really large corpus, but we can still find the popular n-grams, and they are quite different. The top 10 words are ‘ice’, ‘drift’, ‘lead’, ‘indicated’, ‘line’, ‘being’, ‘ship’, ‘large’, ‘slight’, and ‘pack’, and we can clearly detect a very desirable obsession with the sea-ice. Longer phrases include: ‘lead line’, ‘a slight drift’, ‘indicated by the lead’, ‘inches in thickness formed over sounding hole since noon yesterday’.

The voyage of USRC Thetis – April to September 1884

We’ve been running the beta-test of oldWeather Arctic for several weeks now, and we’ve accumulated plenty of completed log pages – that’s log pages that have been transcribed by the three people we need to get reliable results. So it’s time to have a good look at the results we’re getting: Is our new interface collecting the transcriptions properly, and are the transcriptions we’re getting accurate and useful?

This time I’ve tried to show explicitly the link between what we’re doing on the website and the numbers that are going to the science team. The image below shows this for a single log page from USRC Thetis (click on the image for a bigger version).

Log of USRC Thetis for 1884-05-05

The log page for USRC Thetis from May 5th 1884 – as transcribed through oldweather.org. The coloured boxes overlain on the log page are those drawn by the transcribers (blue for weather, red for locations, and grey for dates). The text on the right hand side shows the information collected from the transcriptions.
(Bigger image, Video showing all pages.)


The left hand side of the image shows what we’re doing on oldweather.org – a log page marked-up with the locations of valuable data. (This time I’ve looked only at the dates, positions, and hourly weather observations – the historical events and informal weather records (including ice observations) are just as important but I didn’t have space for them.) As each page is transcribed by at least three people, there are usually three selections for each record. The right hand side shows the values extracted from the transcriptions.

For this page it’s working very well: we’re getting the detailed weather and ship-position information we need. Of course, that’s just one page – we need to do that for every page, and that’s shown in this video, which shows the transcribed data streaming out of the log consistently and accurately. All our hard work transcribing is delivering the detailed weather records the scientists need.

We can also look at the 2718 new weather observations we’ve rescued from the Thetis so far. How do they compare with more recent observations? Were the sailors on this ship careful and accurate observers? To judge this I like to compare the oldWeather observations (red points in the figure below) with modern records.

Observations made by USRC Thetis on her voyage from New York to Smith Sound in summer 1884 (red dots), compared with modern climatologies (black lines).


The top left image shows the route of the ship: from New York up through the Labrador Sea and Baffin Bay and back – a true Arctic voyage. The bottom right image shows that measured air temperatures were typically lower on the voyage in 1884 than the average (climatology) for the last few years – an intriguing result (though there are many possible reasons for it). Top right and bottom left are air pressure and wind speed, these are harder to compare because for pressure and wind we expect bigger differences between an observation (a point value) and a climatology (an average over several years). Rather than going into details I’ll just say that I’m very pleased with these results too; this comparison is exactly what we’d expect from good-quality, useful observations.

So well done USRC Thetis and all who sail with her – both her original crew who took the observations, and the oldweather crew led by Lekiam, Jelliott8 and lollia paolina.

A glowing news report

We’ve made several videos introducing oldWeather and demonstrating the use of the transcriptions produced. But today the Guardian has done even better – producing a very impressive interactive visualisation of the ship movements.

oldWeather Credits Reel

One of the main uses of the weather observations that we are collecting is in new reanalyses – reconstructions of weather and climate over the last few decades or centuries. This week, dozens of scientists working on weather and climate reconstruction are meeting for a workshop on reanalyses and historical weather observations, hosted by the Royal Netherlands Meteorological Institute at De Bildt.

This is an opportunity to tell everybody working in the field just how much we’ve achieved with oldWeather over the last 11 months, so I’m giving a presentation highlighting our results. As you’ll have seen from earlier blog entries, there’s plenty to present – so my biggest challenge is in working out how to give credit to all the project participants: It’s a firm rule in science that you should credit all your collaborators in any project, but there are 9566 people who’ve made a significant contribution to oldWeather (at the last count). So to list them all I’ve borrowed a technique from the movies, and made a credits video – this video is being premiered at the meeting (as part of my talk).

Of course it’s not enough just to have lots of people involved, we’ve also got to generate lots of new scientific results. So I’ll also be showing another video – less detailed, but faster and much more colourful – showing the 841848 new weather observations that we’ve generated.

150 ships

Oldweather is steaming past milestone after milestone, and a few days ago we passed a big one : 150 ships complete. That is; 150 ships, from Acacia to Wonganella, have had all of their log pages for the period transcribed by at least three people. That’s 89,000 pages of new information for climate and historical research.

To mark the completion of 100 logs, we made a movie showing the ships bustling about across the world’s oceans. Rather than updating this with the new information, I thought I’d be a little more ambitious and show the transcribed data in a more comprehensive and interactive format.

I’ve long been an admirer of Google Earth – a geospatial data viewer that’s powerful, easy to use, and, most important, free to download and use. If you haven’t played with it I urge you to give it a try – download a copy and have a look at the satellite’s-eye view of your favourite places. But the real charm of Google Earth is that we can add our own data as overlays. It works very well for following ships; and I’ve made an overlay from the 150 completed oldWeather ships.

So once you’ve got Google Earth, download the overlay and have a look at what you’ve created: select a ship from the list on the left, pick a time using the slider at the top, and click on the markers to see the day’s records for that ship. The user interface takes a bit of getting used to, but with practice you can make your own animations.

I included as many of the transcriptions as I could, but there are some that I haven’t yet managed to convert into this format. So if you can’t find something you know you entered, don’t worry: we haven’t lost it. It will take a little longer, but we will make it all available.

oldWeather in Google Earth

Browsing the oldWeather ship records in Google Earth.

Social Graph of the Royal Navy in WW1

As we said in our recent blog post, Old Weather has been churning through Royal Navy logbooks from World War 1 for long enough now that we can start to extract some interesting stats from the words transcribed by the community.

Social networks are all the rage now, but here at Zooniverse HQ we’ve been wondering what the 90-year-old social graph of Old Weather would look like. We’ll have more to say in the near future about the interactions of people on board the Royal Navy ships from our logs, but what about the ships themselves? When ships pass each other at sea, or meet to exchange supplies, officers and information, they make a note of this in their logs.

This enormous chart shows all of the Old Weather ships in a big grid, highlighting in purple where ships connect to each other. You can look down the chart, or across it, to find the interactions for a given ship. You can see that the HMS Arlanza and the Alsation seem to meet up with quite a few of the other ships of the chart. Both are Armed Merchant Cruisers that cross the busy stretch between the UK and the USA. So is the HMS Motugua, and it too has a fair few interactions with other vessels.

Taking those ships that are often mentioned, we can delve further into their interactions and create arc plots for those vessels. The arc plot below, for the HMS Alsatian, shows that it has encountered 26 ships in the transcriptions made to date. The thickness of the lines connecting vessels indicates the relative number of times that the two ships reference each other. The HMS Moldavia and HMS Patia are fairly well-connected with the Alsatian.

What isn’t shown on the large network plot is that the most mentioned vessel in the Old Weather fleet is the HMS Bee, a river gunboat and a ship that is only 36% complete so far on Old Weather (maybe you could jump aboard and help to complete it?). This ship is not mentioned a great deal by every ship but rather features regularly in the logs of a few vessel in the fleet. The arc plot for the HMS Bee is shown below. The HMS Bee interacts a great deal with the HMS Scarab and the HMS Cricket. all three are gunboats, as is the HMS Gnat. The next step here is to examine the logs and find out when these vessels interacted so much, and why. A blog post of these at a later time.

Finally, for this post, let’s look at the arc plot for the top twenty most-connected vessels in Old Weather so far. These are the ships from the large network plot that connect with the most other ships. These plots can be made for the whole fleet – but they become very large and complex and thus difficult to take value from. This slimmed-down version showing just the top twenty gives you an idea of the ships that are linked to other ships.

This is the kind of simplistic data that can be extracted from your transcriptions of events. So far, only the development team have been looking at this, but the tools are being made available to the historians of Old Weather for further analysis. I’m excited by what they can uncover.

Many of the ships listed in these charts are available on our Old Weather Voyages page, so you can see for yourselves how they interact with each other. You can use that page to read the log entries and see where ships were when they encountered one another. We’re always trying to find new ways for everyone to explore the Old Weather data and if you have any suggestions we’d love to hear them, either here on the blog or via twitter @oldweather.