Improving seasonal climate forecasts

NIWA’s climate gurus are using statistical tools to improve seasonal climate forecasts.

Do meteorologists forecast rain so they get the golf course to themselves? This observation on Larry David’s Curb Your Enthusiasm was backed up by US analysis, which showed that when a 100 per cent chance of rain was forecast in Kansas, it did not rain at all one-third of the time.

Short-term weather forecasts have high accuracy

Despite glitches, short-term weather forecasts, up to 10 days ahead, are one of the great success stories of science and statistics. The average accuracy is about 70 per cent in temperature forecasts and 60 per cent in rainfall forecasts.

Forecasts are based on models of the weather. Data on what is currently happening are fed into a number of models. The forecasts are built from an assessment of what those models say is likely to happen.

A big part of the improvement is due to statistical modelling. Climatologists compare what they forecast with what weather actually does. Consistent inaccuracies can be corrected. This ‘validation’ helps them refine weather models, and helps people interpret them better.

The US National Weather Service found that human interpretation and consensus based on validation analyses of when models were right and wrong improve the accuracy of rain forecasts by about 25 per cent over the computer models. Human judgement improves temperature forecasts by about 10 per cent.

Randomness makes long-term forecasting more difficult

While short-term weather forecasts now have a high degree of reliability, climate forecasts – long-range estimates of what will happen in future seasons – are far less reliable. They are currently accurate only about 33 per cent of the time. Climatologists are turning their statistical tools to the task of improving this form of forecasting.

Dr Nico Fauchereau, who runs NIWA’s Seasonal Forecasting Project, providing forecasts of approaching seasons, says the challenge of long-range forecasting is the randomness inherent in the climate.

“The climate itself hasn’t yet settled on what it will do next season – so we can’t forecast it with the certainty we have for tomorrow’s weather. You can’t predict something where outcomes are influenced by a massive degree of randomness.”

The scale of chaos in weather is breathtaking. The number of molecules interacting in the Earth’s atmosphere has been estimated at 100 tredecillion – that's a 1 followed by 44 zeros. Perfect weather predictions would have to account for all those molecules and solve equations for their interactions all at once. A change in even one of the interactions could change exponentially the interactions of millions of others.

If you could imagine correctly predicting the outcome of every person on Earth tossing a coin 1000 times, you’d still be nowhere near the degree of complexity required to forecast seasons.

Fauchereau says that the chaos is seen as noise in weather data. “For long-range outlooks we’re looking for clear signals of a dominant trend influencing the weather. If there’s no clear signal, noise dominates the data.

“In a strong El Niño – such as the one that is currently affecting the Pacific – the accuracy of forecasts is generally better.

“When there is no clear, large-scale signal, things can be more muddy; the climate is not ‘steered’ in any particular direction, and our confidence in forecasts is generally lower.”

New Zealand’s geography adds to the difficulty

Another factor that makes seasonal forecasting challenging is New Zealand’s geography.

New Zealand lies between the sub-tropics and the mid-latitudes, and the potential for predictability decreases significantly between the tropics and New Zealand’s latitude.

“We could improve our seasonal forecasts if we could move New Zealand a few tens of degrees northwards,” says Fauchereau.

He says New Zealand’s geography makes forecasting, especially long range, even tougher.

“For a small land mass like New Zealand, the distribution of sea surface temperature plays the biggest role in our weather.

“That means our seasons are influenced by a wide range of regional weather signals. Outside the El Niño and La Niña effects, it’s hard to estimate which signal will be most influential on future weather.”

Long-term forecasting is still valuable

Dr Brett Mullan, NIWA’s Principal Scientist, Climate Variability, says that even with this uncertainty long-range forecasts are valued by many sectors, such as farmers, emergency services, regional planners and policymakers.

“They need to know what to expect so they can manage their resources. The type of detail demanded is less than for weather forecasts – customers need to know whether it will be hotter and drier than normal, not the exact temperatures. But accuracy matters even at this level of generalisation.”

Finding the middle ground

The need for accuracy and the implications of errors were amply demonstrated in 2009, when the UK Met Office forecast a ‘barbecue summer’. Unfortunately, the season turned into one of the wettest summers on record, and led to the organisation ending its long-term outlooks.

Mullan says that while NIWA looks at climate on a monthly basis, it issues three-monthly forecasts and believes they are useful as an indication of the expected climate.

“We need an average period of at least three months to predict climate signals with some reliability. The three-month period is a compromise between averaging out the randomness and retaining the maximum amount of skill in the forecasts.

“Our forecast breaks New Zealand into six regions, which is not a fine resolution but is a level at which we’re comfortable forecasting. There are three categories for forecasting temperature and precipitation: below normal, normal, and above normal. Each possibility is given a percentage chance of its occurring.”

For example, at the start of October 2015 NIWA’s climate outlook was that for the next three months there was a 40 per cent chance that temperatures would be near average and a 40–45 per cent chance that they would be below average for all of New Zealand.

There was also a 40 per cent chance that rainfall would be near the seasonal normal and a 40–45 per cent chance of it being below normal in the north and east of the North Island. The north of the South Island had a 35 per cent chance of being near normal and a 40 per cent chance of below normal.

When NIWA’s climate team meets each month to assess the coming season from dynamical model forecasts and statistical models, it looks at validation data for the previous forecasts.

Many models are better than one

“Consensus generally does better than most individual models, and no model does best all the time.”

Mullan explains that forecasts are based on both local (NIWA) models and international models. Some of the models are ‘deterministic’ and assume no randomness is involved, but more and more models these days take account of randomness by running ‘ensembles’ of forecasts – multiple forecasts starting with slightly different initial weather patterns.

“We compare the actual climate – the patterns, and precipitation and temperature – with the forecasts of models.

“We keep track of the weighting and human reasoning that go into each seasonal forecast.”

A record is kept that reveals the extent to which nine different models have correctly estimated previous seasonal and monthly forecasts.

NIWA’s Dr Trevor Carey-Smith heads the Automated Validation System project, an attempt to use these data to weight the results of each model according to its performance in the real world. The system will automatically generate forecasts from each model, modified to account for their accuracy in certain conditions.

“We’re building algorithms that will weigh the influence of the individual models when they are merged to produce a single forecast.

“Not only is validation influencing the accuracy of the individual models, it is ensuring that the most accurate models have a greater influence on the final forecast.

“Automated statistical modelling can look at the conditions and predict from past performance which of the models will be more accurate.”

Carey-Smith’s project will take the statistical forecasts one step further, analysing the success of models on a location basis.

“We provide a regional seasonal forecast, but variability within regions means users apply their own experiences to their locations. If we’re predicting a 75 per cent chance that it will be drier than normal for a region, and their locations are always drier than the region as a whole, their experiences will differ.

“In the future we’d like to use the automated validation to produce forecasts of local variations from the regional forecast.”

The human touch is still needed

Brett Mullan warns of the difficulty of the challenge that the climate team has set itself, and reasserts the value of human intervention.

“The task is made much harder by the fact that the core models themselves are always being changed to improve their accuracy. That means that statistics looking at the past performance may no longer be relevant.

“Statistical models and automated validation are going to improve the accuracy of seasonal forecasts in normal conditions and with strong climate signals like El Niños. But at the moment it’s humans who are best at recognising and responding to the weird and unusual patterns. We still do random best.”

Mullan says the introduction of validation and probabilities to the forecast has been a major step for consumers of weather information.

“The probability of the outcomes reflects the inherently uncertain state of the long-term climate. Yet there is enough information in the forecast odds to add value to regular economic decisions.

“When certainty is high the signal is strong – and people ought to pay attention to the possibility of droughts or high rainfall. Over the course of several seasons, this knowledge and use of climatic odds translate into dollars saved.

“Three-month outlooks are designed to help people understand developing situations against the normal seasonal outcomes – while being wary of unexpected outcomes,” he says.

Meet FitzRoy, the weather ‘man’ behind it all

The NIWA supercomputer that crunches seasonal data is named after Robert FitzRoy, a Governor of New Zealand and captain of the Beagle on Charles Darwin’s voyage that led to his theory of natural selection.

On returning to Britain, FitzRoy’s fascination with the weather on sailing voyages led to a late career change, at 54 years old, to ‘Meteorological Statist to the Board of Trade’. His new department was intended to design wind charts for navigation.

FitzRoy was inspired to develop charts for what he called “forecasting the weather” by the sinking of the boat Royal Charter in a terrible storm in 1859. Land stations were established to use the new telegraph to transmit to him daily reports of weather at set times. FitzRoy assessed the reports to judge whether to issue storm warnings for ports. When he realised that the sorts of data he was collecting had wider usage, he started turning them into the first-ever daily weather forecasts, published in The Times in 1861.

While the forecasts quickly became massively popular, they were also lampooned for their regular inaccuracy. In May 1862 The Age wrote ahead of the famous Derby horse race that, as Admiral FitzRoy had forecast moderate to fresh winds with some showers, it would be “a remarkably fine day, and … the umbrellas might be left behind”.

There were more serious challenges to FitzRoy’s forecasts. Some politicians complained about the cost of the telegraphs. The scientific community was sceptical of his methods, claiming they lacked a coherent theory. The maritime sector was supportive, but others begrudged workdays lost to mistaken storm forecasts.

In battle with critics, FitzRoy worked hard to decode the British weather. He published a book and gave lectures, but in 1865 he retired, exhausted and beset by depression. He took his own life that year.