26 January 2010

Hot on the Trail of the IPCC Mystery Graph

UPDATE:

I think I have figured it out.

1. The Figure above does not appear in Muir-Wood.

2. A different figure, 1.5 from the SOD, was criticized by reviewers. Figure 1.5 was a scatter plot of temperature anomaly and normalized damage 1950-2005, shown below.


3. In response, the IPCC instead chose to make its own Figure SM 1-1 above at the top of this post, which plotted only 1970-2005 and smoothed the data.

4. The original Figure 1.5 from the SOD is ultimately included in the published version of Muir-Wood et al. in 2008, exactly as it appeared in the SOD, with this explanation:
Without fully controlling for other factors that could affect the trend in losses, we can not draw any firm conclusions about the role of climate change in loss trends. In addition, any conclusions about a relationship between global warming and disaster losses are complicated by the sensitivity of statistical results to a few high-loss data points, the short historical loss record, and the limitations of the normalization methodology.
Hypothesis: The IPCC created a graph that did not exist in the peer reviewed literature or in the grey literature to suggest a relationship between increasing temperatures and rising disaster costs.

END UPDATE
In the comments Ian does some brilliant detective work to locate the origins of the above mysterious and misleading graph from the IPCC AR4 WGII Chapter 1.
I spent some time parsing through the FOD, SOD and related comments for this portion of Chapter 1 of the WGII report.

The graph that Hu McCulloch asks about, and which Roger criticizes above, initially appeared as part of the text in the SOD as figure 1.5. It was a scatter graph, with a “Fitted Values” trend line drawn in. At that point, the source was being cited as “Miller et al. 2006” rather than Muir-Wood; I assume, however, that it is the same source (the full reference was: “Miller, S., R. Muir-Wood, et al. 2006: Weather related catastrophe loss trends and the impact of climate change,. In lit. (to be circulated prior to publication).”) (See p. 91 of Ch. 1 of the SOD) [PIELKE COMMENT: YES]

The use of the “Miller” paper drew a number of comments from the expert reviewers. One of those experts was Annick Douguedroit of University de Provence, who commented (http://ipcc-wg2.gov/AR4/SOD_COMMS/Ch01_SOD_Expert.pdf at p. 122)
“Fig 1,5 is not reliable from a statistical point of view because the significant trend is pulled upward by "outliers" (especially 3 points with losses >100000 ) which provoque a pseudo-significancy as it is suggested by the authors themselves in lines 18-21 [of the SOD] "Removing......entirely". So I propose "Since 1970 the global normalized results do not show any statically significant correlationn with global temperatures." and to remove the end of the paragraph and the figure 1,5 because it can mislead a reader not familiar with correlation."
The response from the “Writing Team” was:
“Figure moved to Supplementary Figure and employed a different plot that smoothes catastrophe losses and shows these alongside temperature. After smoothing (that thereby removes the peaks noted) the correlation remains. The text now provides a balanced commentary on this.”
So, it is clear that the graph seen in the supplementary materials to Chapter 1 of WGII, was the creation of “Writing Team”. Perhaps Roger can answer whether the original graph (as shown in the SOD) was the work of Muir-Wood. [PIELKE RESPONSE: SEE UPDATE ABOVE. I FIGURED IT OUT.]

Concerns were also raised by Indur Goklany (US Dept of the Interior), Francis Zwiers (Can. Centre for Climate Modelling and Analysis) and Nathan Gillet (University of East Anglia).

Zwiers commented (pp. 121-22):
“I’m wondering if too much space is devoted to Miller, given the inference one draws from Fig 1.5 is senstive to the inclusion of individual outliers (as pointed out in the text) and that it is acknowledged that early data are incomplete. Some additional comment on data quality, beyond just completeness, is probably in order (I'm not expert, but this type of data would presumably be influenced by all kinds of factors, including varying political influences and changes in reporting practices, that might confound any climate signal)."
To which the response was:
“Agreed – will restate the conclusions of this work judiciously to summarize what is revealed and what is uncertain.”
Gillett was concerned about the statistical underpinnings (p. 122):
“Is the statistically significant correlation purely a result of the trend in both series? Does the correlation remain statistically significant if both are de-trended? If not, then this merely tells us that both series contain a trend. More fundamentally, why correlate losses with global temperature? Some justification is needed.”
To this, the Writing Team responded:
“Losses can be correlated with year and also with global temperatures. The correlation with T is a function of both series containing trends with time over this period.”
I’ll leave it to the stats types to argue over whether the answer was actually responsive.

12 comments:

  1. There are also problems with the citing. Muir-Wood et al. 2006 and Miller et al. 2008 are very different papers, by the same author team.

    The SM 1-1 Figure is sourced to Muir-Wood but does not originate in that paper, but in Miller. That is, Muir-Wood does not discuss temperature. However, Miller, which does discuss temperature, did not meet the publication deadline. Yet the Figure SM 1-1 was nonetheless included (and remade by IPCC independent of Miller's analysis) and then misreferenced back to Muir-Wood to get around the publication deadline for IPCC.

    Again, not good.

    ReplyDelete
  2. The wealth in the world has increased significantly betweeb 1970 and 2005. That losses have increased in that same time is to be expected.

    ReplyDelete
  3. I certainly missed the significance of the citation of Miller.

    In re-examining Chapter 1, I noticed that Muir-Wood is listed as a "Contributing Author". I confess I've not fully worked out the hierarchy here - but shouldn't he have said something about this inclusion (and the use of his work)?

    ReplyDelete
  4. I think it is worth to remember the findings of other scholars in this field. The following studies (all peer-reviewed!) did not find any significant trend after disaster losses were normalised:

    Hurricane losses in the US (Pielke Jr. and Landsea, 1998; Pielke Jr. and Downton, 2000; Pielke Jr. et al., 2008), tornadoes in the US (Brooks and Doswell, 2001), hurricane losses in the Caribbean region (Pielke Jr. et al., 2003), weather extremes in the US (Changnon et al., 2000; Changnon, 2003), flood and windstorm losses in Europe (Barredo, 2009, 2010), tropical cyclones in India (Raghavan and Rajesh, 2003), and weather-driven disasters in Australia (Crompton and McAneney, 2008).

    All of these studies found no significant trends of losses after historical events were normalised to current societal conditions.

    ReplyDelete
  5. "After smoothing ... the correlation remains"
    Indeed. This is a mathematical truism. It is not proof of anything (except the statistical ineptitude of the commenter).

    Smoothing does not remove the outliers; it just smears them out. This response to the reviewer is wrong -- in the mathematical sense of the word.

    ReplyDelete
  6. Roger and Richard, the IPCC comment about smoothing and correlation is embarrassing to say the least.

    ReplyDelete
  7. I believe the main complaint must be the incomplete coverage of other literature. The Muir-Wood / Miller study shows impacts on a global, aggregate level, while much evidence has also been collected for individual weather extremes at national and continental scales. Those studies show no impact of anthropogenic climate change. Recent papers by Barredo on European losses, discussed by Roger, and other studies show this. As I commented before (and as 4AR WGII reviewer), and commented by Joey (4), both IPCC and Stern should have done a more comprehensive coverage of the literature. Mentioning only the Miller study that find some signal seems to imply selective quoting.

    ReplyDelete
  8. http://www.timesonline.co.uk/tol/news/environment/article7003622.ecehttp://www.timesonline.co.uk/tol/news/environment/article7003622.ece In the London Times today.

    ReplyDelete
  9. There is further evidence of increasingly dubious foundations of the IPCC's proclamations - and the exercise of their much vaunted "peer review" process.

    I have done an analysis of one Lead Author's responses to 292 Reviewer comments on the Second Order Draft of Chapter 6 [based on AR4SOR_BatchAB_Ch06-KRB-1stAug.doc which I had found in the Climategate files] that you might be interested in viewing.

    "The climate change game ... Monopoly: the IPCC version"

    http://hro001.wordpress.com/2010/01/18/the-climate-change-game-monopoly-the-ipcc-version/

    From the almost daily disclosures that have been surfacing of late, along with your posts, I'm beginning to wonder if there is any chapter in this whole body of work which has not been tainted by this polluted process.

    ReplyDelete
  10. That figure appears in the supplementary material with the caveat that they cannot and should not be read in isolation from the chapter.

    In the chapter it says that the 2004 and 2005 losses are "remarkable"

    ReplyDelete
  11. The bias is so obvious. If you look only at the smoothed loss line, it seems to show three 12 year periods of flat, linear ris and flat. A serious scientist would have pondered over this, butnot one driven by AGW forcings

    ReplyDelete
  12. That figure appears in the supplementary material with the caveat that they cannot and should not be read in isolation from the chapter.

    ReplyDelete