Consensus building –
Replication isn’t only about repeating the exact same study.
Hurricanes are fed by warm ocean waters, so it’s expected that we’ll see larger, more dangerous storms as the planet continues to heat up due to climate change. These storms also have a major economic impact — Katrina is thought to have knocked a full percentage off the US ‘GDP in 2005. As such, we might expect to see hurricane-driven economic damages to be increasing over time.
But there’s a lot of statistical noise in that system. Many storms miss the US entirely, and others miss the major urban centers and damage sparsely populated areas. The growth of wealth over time, as well as things like changing business codes, also complicate the analysis. As a result, detecting any change in damages is extremely difficult, and studies have come to very different conclusions about whether there’s a trend or not.
But a new study suggests that the most powerful storms are in fact increasing. And perhaps just as significantly, its findings are consistent with a study of natural disasters published just a month ago, even though the two used very different methods.
Hurricanes and other disasters
We went over all the reasons that measuring trends in natural disasters is complicated when wecovered a study on the topiclast month. Rather than reiterate that, we’re going to focus on contrasting the methods that paper used with those that this week’s study relied on.
The study published last month was on weather-related natural disasters in general; it included hurricanes but also events like droughts and heat waves. It adjusted for the changing demographics of locations struck by disasters by accounting for things like the population and GDP of the areas affected by the disaster. These showed a clear trend when it came to the most costly disasters: there are more of them than a century ago, and some are costing more than any we saw earlier, even after economic growth is accounted for.
This week’s study, in contrast, focuses on hurricanes. Again, it compensates for things like the local population and its economic activity. But it does so by a distinct method, combining data on the storm’s location and its damages to calculate what the authors term an “area of total destruction.” This measure accounts for the wealth of the area struck by the hurricane by normalizing it to a ratio calculated by comparing the value of what was actually destroyed to what potentially could have been destroyed. This was calculated for a 10, 00 0-square-kilometer area surrounding the storm’s landfall location, based on the size of recent large storms.
Thus, although the two studies are similar in terms of the analysis they want to perform, their particular approaches are distinct.
Despite this, the result of the new analysis is similar to that of the earlier one. Two trends were apparent. The first is that the overall destructiveness of hurricanes has been climbing slowly over the last century. The second trend is that the most destructive storms are increasing in destructiveness faster than the less harmful ones. For example, moderately destructive storms have increased in destructiveness by about 1.5-fold from 1900 to the 2010 s. By contrast, the most destructive storms have increased by 3.3-fold.
The authors also went back and re-analyzed the data used to conclude that there was no statistically significant overall trend when economic growth was adjusted for using a different method. Even in this data, they were able to see that the most destructive events had increased by a statistically significant amount — 3.2-fold in this case, not significantly different from what they saw in their own analysis.
It also agrees with the earlier analysis, which also saw that changes in the cost of natural disasters were being driven by the largest events.
Replication and independence
The authors acknowledge that their work is a first attempt and could be improved upon significantly. The study simply takes a large area surrounding the hurricane’s site of landfall and assumes that this area is big enough to have a GDP similar to that of the US as a whole. It’s possible to get better regional measures of GDP and to track the path of a storm and the radius of its winds. But that sort of data isn’t available for the earlier storms in this data set, which may limit the analysis to a shorter time period.
But even in its current form, the study is interesting because of how it fits in with the other literature on the topic . It sees the same trends in hurricanes that people have found for weather-driven disasters in general. And the same effect can be seen if a different method is used to compensate for economic growth. All of which helps provide a bit more confidence in the results.
The ability to produce consistent results using different methods is often a critical way that scientists end up replicating a finding and occurs far more often than we see in direct repetitions of the same experiments. Which is why the focus on directly repeating identical experiments that has accompanied the talk of a “replication crisis” seems somewhat off target. In many ways, repeating an experiment and not getting the same result tells us far less about what’s going on than having looked at the same problem from multiple angles.