30 August 2010

Report of the IAC Review of the IPCC

The InterAcademy Council Review of the IPCC has been released. The report is remarkably hard hitting with constructive and far-reaching consequences.  In the report's own words:
If adopted in their entirety, the measures recommended in this report would fundamentally reform IPCC’s management structure while enhancing its ability to conduct an authoritative assessment.
It is an excellent, thoughtful report.  While the report focuses on procedural questions and does not address any questions of scientific content, its recommendations have far-reaching substantive implications, such as for how to deal with uncertainty.  The report also directly addresses difficult subjects such as conflict of interest, policy advocacy and tenure of the IPCC chairman.

This post is simply a summary of the report's recommendations, and I will soon follow it with a bit more analysis.  Here then are the report's "key recommendations" that are highlighted in the Executive Summary, follwed further below by the recommendations found in the body of the text:
Governance and Management

The IPCC should establish an Executive Committee to act on its behalf between Plenary sessions. The membership of the Committee should include the IPCC Chair, the Working Group Co-chairs, the senior member of the Secretariat, and 3 independent members, including some from outside of the climate community. Members would be elected by the Plenary and serve until their successors are in place.

The IPCC should elect an Executive Director to lead the Secretariat and handle day-to-day operations of the organization. The term of this senior scientist should be limited to the timeframe of one assessment.

Review Process

The IPCC should encourage Review Editors to fully exercise their authority to ensure that reviewers’ comments are adequately considered by the authors and that genuine controversies are adequately reflected in the report.

The IPCC should adopt a more targeted and effective process for responding to reviewer comments. In such a process, Review Editors would prepare a written summary of the most significant issues raised by reviewers shortly after review comments have been received. Authors would be required to provide detailed written responses to the most significant review issues identified by the Review Editors, abbreviated responses to all non-editorial comments, and no written responses to editorial comments.

Characterizing and Communicating Uncertainty

All Working Groups should use the qualitative level-of-understanding scale in their Summary for Policy Makers and Technical Summary, as suggested in IPCC’s uncertainty guidance for the Fourth Assessment Report. This scale may be supplemented by a quantitative probability scale, if appropriate.

Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs).

Communications

The IPCC should complete and implement a communications strategy that emphasizes transparency, rapid and thoughtful responses, and relevance to stakeholders, and which includes guidelines about who can speak on behalf of IPCC and how to represent the organization appropriately.

Here are additional recommendations found in the body of the report:
The IPCC should make the process and criteria for selecting participants for scoping meetings more transparent.
The IPCC should establish a formal set of criteria and processes for selecting Coordinating Lead Authors and Lead Authors.

The IPCC should make every effort to engage local experts on the author teams of the regional chapters of the Working Group II report, but should also engage experts from countries outside of the region when they can provide an essential contribution to the assessment.

The IPCC should strengthen and enforce its procedure for the use of unpublished and non-peer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report.

Lead Authors should explicitly document that a range of scientific viewpoints has been considered, and Coordinating Lead Authors and Review Editors should satisfy themselves that due consideration was given to properly documented alternative views.

The IPCC should adopt a more targeted and effective process for responding to reviewer comments. In such a process, Review Editors would prepare a written summary of the most significant issues raised by reviewers shortly after review comments have been received. Authors would be required to provide detailed written responses to the most significant review issues identified by the Review Editors, abbreviated responses to all non-editorial comments, and no written responses to editorial comments.

The IPCC should encourage Review Editors to fully exercise their authority to ensure that reviewers’ comments are adequately considered by the authors and that genuine controversies are adequately reflected in the report.

The IPCC should revise its process for the approval of the Summary for Policy Makers so that governments provide written comments prior to the Plenary.

All Working Groups should use the qualitative level-of-understanding scale in their Summary for Policy Makers and Technical Summary, as suggested in IPCC’s uncertainty guidance for the Fourth Assessment Report. This scale may be supplemented by a quantitative probability scale, if appropriate.

Chapter Lead Authors should provide a traceable account of how they arrived at their ratings for level of scientific understanding and likelihood that an outcome will occur.

Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs).

The confidence scale should not be used to assign subjective probabilities to ill-defined outcomes.

The likelihood scale should be stated in terms of probabilities (numbers) in addition to words to improve understanding of uncertainty.

Where practical, formal expert elicitation procedures should be used to obtain subjective probabilities for key results.

The IPCC should establish an Executive Committee to act on its behalf between Plenary sessions. The membership of the Committee should include the IPCC Chair, the Working Group Co-chairs, the senior member of the Secretariat, and 3 independent members, including some from outside of the climate community. Members would be elected by the Plenary and serve until their successors are in place.

The term of the IPCC Chair should be limited to the timeframe of one assessment.

The IPCC should develop and adopt formal qualifications and formally articulate the roles and responsibilities for all Bureau members, including the IPCC Chair, to ensure that they have both the highest scholarly qualifications and proven leadership skills.

The terms of the Working Group Co-chairs should be limited to the timeframe of one assessment.

The IPCC should redefine the responsibilities of key Secretariat positions both to improve efficiency and to allow for any future senior appointments.

The IPCC should elect an Executive Director to lead the Secretariat and handle day-to-day operations of the organization. The term of this senior scientist should be limited to the timeframe of one assessment.

The IPCC should develop and adopt a rigorous conflict of interest policy that applies to all individuals directly involved in the preparation of IPCC reports, including senior IPCC leadership (IPCC Chair and Vice Chairs), authors with responsibilities for report content (i.e., Working Group Co-chairs, Coordinating Lead Authors, and Lead Authors), Review Editors, and technical staff directly involved in report preparation (e.g., staff of Technical Support Units and the IPCC Secretariat).

The IPCC should complete and implement a communications strategy that emphasizes transparency, rapid and thoughtful responses, and relevance to stakeholders, and which includes guidelines about who can speak on behalf of IPCC and how to represent the organization appropriately.

19 comments:

  1. Looks like there is going to be a lot of demand for statisticians in the proposed IPCC.

    ReplyDelete
  2. The report is indeed good, as one would expect from the panel.

    I would have made more of the selection of authors.

    I would have called for a permanent regulator of the IPCC (that is, a continuous IAC rather than an ad hoc one), franchising of the IPCC brand, and alternative assessments.

    ReplyDelete
  3. In matters scientific I am a sceptic. In matters political (especially UN) I am a cynic.

    Let's see who ends up as an "independent committee member". I hear Al Gore has some free time and is a Nobel prize winner to boot.

    I am sure James (Nazi coal trains) Hansen could find some time to devote to extracurricular activities such as this.

    ReplyDelete
  4. The IPCC seems to be rearranging the deck chairs. The WP has an article concerning the sinking of the climate Titanic: http://www.washingtonpost.com/wp-dyn/content/article/2010/08/29/AR2010082903699.html?hpid=topnews

    ReplyDelete
  5. This is the lead paragraph from the IAC webpage:

    The process used by the Intergovernmental Panel on Climate Change to produce its periodic assessment reports has been successful overall, but IPCC needs to fundamentally reform its management structure and strengthen its procedures to handle ever larger and increasingly complex climate assessments as well as the more intense public scrutiny coming from a world grappling with how best to respond to climate change, says a new report from the InterAcademy Council (IAC), an Amsterdam-based organization of the world’s science academies.

    -----

    AGW and by connection the IPCC have a unique position in science due to the policy implications and the IPCC needs to keep up with the demands of that. From reading that intro page on the IAC site, it all seems reasonable and fair to me. So far the IPCC has done a decent job, but it needs to professionalize. It's only staffers so far are people who do the logistics of publishing. These reforms will lead to a much larger and more expensive organization. I agree with Roger that it should have strong conflict-of-interest guidelines, and a professionalizing process should lead to more consistent following of guidelines that already exist, or new ones that are instituted.

    I expect that some commentators on this report will portray it as saying the IPCC reports have been faulty or the organization corrupt. What this report represents to me is the need to take the IPCC to the next level.

    ReplyDelete
  6. Frankly the IAC and the IPCC are too inbred and intertwined to result in a credible report.
    Why is big science, supposedly so enlightened and sacrosanct, so unable to do an arms length review of climate science?
    Why so much effort is going to avoiding what is inevitably going to happen anyway? Reviews- real reviews- will be done. Real audits as well. Big science is no more immune to this than is any other part of society. The childish efforts at blaming the skeptics and performing shoddy whitewashes only underscores the need.
    Does instiutional science really hold itself out as somehow immune to realities of disclosure, audit, and accountability?
    The IAC may be a step in the right direction, but if so, it is tentative and minimally incremental at best.

    ReplyDelete
  7. Authors would be required to provide detailed written responses to the most significant review issues identified by the Review Editors ...

    Roger, how does this not give ample wiggle room to an editor who - for example - ignores a comment that hurricanes are not causing more damage? Surely the editor could classify that comment as something other than a "significant" issue.

    I must confess to being skeptical that this will increase transparency.

    ReplyDelete
  8. The report is fantastic, far beyond my expectations.

    Now the IPCC has to implement it. That is always the tough part.

    ReplyDelete
  9. All these "IPCC shoulds...." Great suggestions, but most will unlikely be heeded, just like the guidelines that are already in place. If they DID heed all this advice, the next IPPC report would have to say, essentially, that we don't know much about climate change, IMHO.

    I really disagree with this one:

    "The likelihood scale should be stated in terms of probabilities (numbers) in addition to words to improve understanding of uncertainty. "

    I don't think numbers should EVER be used to communicate qualitative information (weight of opinion?). Numbers are quantitative, period, and should only be used for quantitative statements.

    ReplyDelete
  10. "If adopted in their entirety, the measures recommended in this report would fundamentally reform IPCC’s management structure while enhancing its ability to conduct an authoritative assessment. "


    This is another way of saying that the IPCC's current assessments are not authoritative.

    ReplyDelete
  11. Roger - you are quoted in the following story:

    http://www.telegraph.co.uk/earth/environment/climatechange/7972034/IPCC-told-to-stop-lobbying-and-restrict-role-to-explaining-climate-science.html

    One line, but still... ;-)

    ReplyDelete
  12. From memory. The system of consensus is fundamentally corrupt. Like other parts of the UN (e.g. the WHO), there is undue corporate influence.

    Pachauri was simultaneously head of the IPCC and an oil executive.

    Pachauri was on the Board of Directors of the Indian Oil Corporation (January 1999 to September 2003

    On 20 April 2002, Pachauri was elected Chairman of the Intergovernmental Panel on Climate Change, a United Nations panel established by the WMO and UNEP to assess information relevant for understanding climate change.

    http://en.wikipedia.org/wiki/Rajendra_K._Pachauri

    The world's 'greatest living scientist' reveals the problem.

    James Hansen

    Governments today, instead, talk of "cap-and-trade with offsets", a system rigged by big banks and fossil fuel interests. Cap-and-trade invites corruption. Worse, it is ineffectual, assuring continued fossil fuel addiction to the last drop and environmental catastrophe.

    http://www.guardian.co.uk/environment/cif-green/2010/aug/26/james-hansen-climate-change?showallcomments=true#end-of-comments


    The recommendations are merely a list of the more obvious deliberate (IMO) weaknesses in the system which have been highlighted before.

    ReplyDelete
  13. With few exceptions, it seems to me that this report is simply saying that the IPCC should behave in the way it was always supposed to have been behaving in the first place.

    The real question is why we should expect that the IPCC will diligently follow these new guidelines, when they have so blatantly ignored their own procedures in the past. After all, they were never supposed to quote WWF pamphlets, but they did....

    ReplyDelete
  14. "Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event (e.g., based on measurement, expert judgment, and/or model runs)."

    " . . . and/or model runs)."

    More results produced by computer software that has not yet been Independently Verified. And not Verified means that the numbers have not yet been shown to be those that are expected to be produced by the software.

    Validation is a whole nother issue and not to be confused with Verification. Verification must always precede Validation. Validation demonstrates that the numbers are in accord with, have sufficient fidelity to, physical reality.

    ReplyDelete
  15. I fully expect that the UN's work on science will continue to be every bit as honest and balanced as the work the UN does on human rights.

    ReplyDelete
  16. No one has clearly explained why we need another IPCC report; I think we have enough to go on in terms of information to have the policy debate we need to have.

    At the BBC/KGNU climate change panel discussion in June in Boulder, I believe it was Caspar Amman who said that his research was rushed to meet IPCC deadlines.

    I'm still not clear why everyone needs to go through this again, except for bureaucratic inertia. Could someone describe exactly what we hope to get out of this exercise and why it is worth the effort? Perhaps instead we should have an international panel of economic experts telling us what the conditions and projections of our economies are..

    ReplyDelete
  17. Sharon F.
    In my ignorance, I had wondered why, if the crisis was almost upon us, we could wait until 2014 for the next report.

    I think that the need for continuing the IPCC is driven by its sponsors' need to drive (NOT INFORM) policy.

    I think Dr. Spencer is right, it should be dropped since it is unavoidably an advocate.

    That it could ever have been an "honest broker" as Richard Tol suggests seems unlikely.

    Not yet having read Roger's book, I am skeptical that an "honest broker" could exist in any controversial area. I would be happy enough with a broker with admitted biases.

    ReplyDelete
  18. Sharon F. said... 16

    "No one has clearly explained why we need another IPCC report;"

    SOP in politics
    1) Identify potential problem
    2) Have blue ribbon panel study problem
    3) Politico's run around waving report and proclaiming urgent need for action
    4) Action occurs or go back to step 2

    ReplyDelete
  19. Until I heard from Dr. Amman at the panel discussion, I had no idea that the rush to provide info to IPCC could actuallly reduce the quality of the information.

    I think that aspect, albeit little discussed, is very important in preparing scientific information for policy makers- it should be better than average information if people are to make serious financial decisions on the basis of that information.

    ReplyDelete