21 July 2011

The Policy Advisor's Dilemma

My latest column for Bridges is out.  In it I discuss a challenge faced by the policy advisor -- how to be close enough to a decision maker to participate in decision making, but not so close that independence is lost.

I draw on the work of my friend and colleague Eva Lövbrand who has looked in depth at the role of the EU ADAM project in relation to the needs of decision makers. I compare her analysis of ADAM's research in decision making with the role of intelligence in the decision to go to war in Iraq (an issue I discuss at length in The Honest Broker).

In my column I ask:
What is the difference between the case of WMDs, where policy analysis was provided in response to the stated needs of decision makers, and the case of ADAM in which policy analysis was similarly provided in response to the stated needs of decision makers?
For the answer, please have a look at the column, and please feel welcome to come back here with comments and critique.

Reference:  Lövbrand, E. (2011). Co-producing European climate science and policy. A cautionary note on the funding and making of useful knowledge. Science and Public Policy 38(3): 225-236.

26 comments:

  1. Trimming the truth in order to be popular with the in crowd is always tempting. But we teach our kids that their integrity is more important and that succumbing to the siren song brings disaster.

    ReplyDelete
  2. Very interesting analyses, both Eva's and yours. Having been part of the ADAM project myself, I'd like to add two observations.

    First, the project was funded by the EU's Research DG (Directorate-General). While expected policy impact was a criterion when evaluating the project proposal, the purpose of the project was to do (policy-relevant) scientific research, not to give policy advice. To ensure policy relevance we always invited representatives of other EU DGs, including those responsible for designing and implementing climate policy. I remember how one of these people, after a series of scientific presentations at an annual meeting of ADAM, compared ADAM with a group of people travelling through space for three years cut off from any contact with earth, trying to come up with a solution to the problems the earth faced when they left. When they returned three years later they presented the perfect solution to the problems of three years ago, but unfortunately it had become irrelevant to the problems of today.

    Second, ADAM was about both adaptation and mitigation, but for the first two years of the project the EU policy practitioners with whom we interacted had very little interest in adaptation. Then suddenly the political context changed and they became very interested in adaptation. They turned to ADAM to look for any interim results, but found they weren't as policy-relevant as they had hoped. There was no acknowledgement that their previous lack of interest in interacting with the ADAM adaptation researchers could be one of the reasons.

    I think the lessons from ADAM aren't only about the inherent tensions between scientific research and policy advice, as you and Eva describe them. What struck me was the mismatch in expectations between researchers and stakeholders, and in the notion of ownership of the project. As far as DG Environment was concerned ADAM initially didn't deliver useful results, that is, arguments that could support their policies. Of course, as a scientific project ADAM wasn't designed to do that in the first place, but it was then pushed to move into that direction. In the end it did deliver some policy-relevant results, but this was at the expense of overall project coherence.

    So while Eva and you make some very good points in your analyses, you should recognise that ADAM was designed and funded as a research project, to run for three years at some distance from day-to.day policy. Policy people in another part of the EU then perceived that distance as a gap, and the responsibility for bridging that gap was put squarely with the researchers.

    ReplyDelete
  3. -2-Richard

    Thanks for this comment ... the distinction between "policy-relevant scientific research" and "policy advice" is often terribly fuzzy, and I think that is the case with ADAM. Certainly there was a mismatch, as you describe, in expectations among various participants. No doubt this is in part due to a general unwillingness to actually come to terms by what is meant by "policy relevant scientific research."

    Thanks!

    ReplyDelete
  4. The most important point in your article is the fact the political decision had already been made by DG. Analysis following a political decision is simply a means to justify the decision already made. An article in the Spring Issue of Regulations Magazine by Shapiro and Borie-Holtz "Lesson from NJ" supports the idea "that many rulemaking decisions are set before a proposed rule is even issued." And once set no contrary evidence can alter its course.
    The dilemma you describe emerges from the fact that independent analysis is a threat to political/rent seeking interests behind the rule making.

    ReplyDelete
  5. Based on the Bridges article, it sounds like ADAM was seen by the decision makers as a public relations firm - there to help them sell their program. Of course, this is the way politics works, with its think tanks, committee staffs, administration departments, etc.

    There's a big difference between experts giving advice on developing policy and experts giving advice on how to justify existing policy.

    ReplyDelete
  6. -4- and -5-

    That's why I emphasised that ADAM was a research project, funded by DG Research and to be conducted at some distance from DG Environment, which at the time was responsible for climate policy.

    Based on earlier studies DG Environment certainly had an idea of the kind of climate policies they wanted to put forward. They were hoping ADAM would provide more justification for their policies. It's not that ADAM argued otherwise, but its focus was more on developing robust methods for multi-stakeholder policy evaluation. As far as DG Environment was concerned, this wasn't relevant.

    ADAM was set up as a research project by DG Research. It's unclear to me what makes Mark B think it was as a public relations firm helping to sell policies.

    ReplyDelete
  7. Anytime you deviate from objective scientific evaluation to supporting a specific goal or agenda, you are no longer a science advisor.

    You are a science strategist.

    I'm sure this is covered in the definitions of what is and what is not an honest broker, but the evaluation of real world 'science advisors' as actually being 'science strategists' is one which isn't performed.

    Probably there is no way to objectively do this, except of course there are many public instances of 'science strategists' such as Dr. Hansen of GISS.

    ReplyDelete
  8. -7-Khan

    Thanks, I do indeed discuss a parallel notion in The Honest Broker under the phrase "issue advocate" which can be overt or stealth. The stealth kind is far more problematic.

    ReplyDelete
  9. -6-

    There may be a difference between what the program was set up to do and what decision makers wanted it to do.

    "Lövbrand explains that the officials "hoped that the ADAM research might lend support to the lowest of the [emissions scenarios] considered by the IPCC""

    ReplyDelete
  10. [From your Bridges article]:

    "Having already committed to a policy proposal, the policy makers had little desire to see the project's varied policy analyses. Instead, they needed information that would be useful in advocating the course of action to which they had already made a commitment."

    To some extent, such a description is equally apt when considering the relationship of the IPCC to the UNFCCC, is it not?

    For all Pachauri's puffery about "policy-relevant but not policy-precriptive", the IPCC is - in effect - constrained by the expectations of what he has called the IPCC's "main customer" which is how he has described the relationship between the IPCC and the UNFCCC.

    This certainly does not excuse all the known shortcomings of the IPCC; but, IMHO such a dynamic (for want of a better word) does go some way towards explaining that which might otherwise be considered inexplicable!

    ReplyDelete
  11. -10-hr001

    Yes, I agree. In the first IPCC WGIII was titled "Policy Options" (1991), after Rio in 1992 "options" were no longer needed, and the title of WGII has evolved accordingly, such that by 2007 it was focused on carbon trading and Kyoto.

    More here:
    http://cstpr.colorado.edu/prometheus/?p=3661

    ReplyDelete
  12. Thanks for the pointer to the additional history, Roger! As a newcomer to the "climate wars" (a few weeks BC [Before Climategate]), my "due diligence" (for the most part) only went back as far as AR4. And here I thought that all their advocacy efforts (apart from the hockey-stick!)were a more recent development!

    Seems to me that perhaps rather than placing itself on a pedestal, 'twould have been a far, far better thing had the IPCC decided to quit while it was still ahead (1991).

    This "gold standard" has become very tarnished in the intervening years. And the IPCC's more recent failure to grasp the lifeline handed by the IAC, does not bode well for its future, IMHO.

    ReplyDelete
  13. -11-

    Correction: In 1990 (not 1991) Working Group III was titled 'Response Strategies', not 'Policy Options'. See here. It covers both adaptation and mitigation.

    I am not aware that agreement on the UNFCCC in Rio meant that this Working Group was no longer needed. As you can see from the above link, the contents of the 1990 Working Group III report were not in a form that could have served as direct input into the UNFCCC negotiations.

    On the contrary, once the text of the UNFCCC was agreed and it entered into force in 1994, governments wanted more information on the kind of policies and measures they could implement to mitigate climate change, and in response to that need, the mandate of Working Group III was changed to cover only mitigation. Adaptation was moved to Working Group II, although the emphasis of that Working Group remained on impacts, not adaptation.

    -12-

    This isn't meant as sarcasm: how about having a look at the IPCC website if you want to find out more about the IPCC?

    ReplyDelete
  14. -13-rjtklein

    Thanks for the correction, yes "response strategies"

    Of course 1990 was not prepared for FCCC since the FCCC was only adopted in 1992! Since that time there has been a steady narrowing of the IPCC WGIII agenda to provide input to the official negotiations.

    Just as with ADAM one could argue that this was entirely appropriate, but obviously it means that alternatives to the FCCC were largely ignored by IPCC.

    ReplyDelete
  15. -14- Roger

    "Since that time there has been a steady narrowing of the IPCC WGIII agenda to provide input to the official negotiations."

    That's an interesting statement, although without additional information I wouldn't be able - off the top of my head - to say if that's true or not.

    It wouldn't be surprising if it were true because it's governments that negotiate the implementation of the UNFCCC and it's governments that agree on the issues to be assessed by the IPCC. On the other hand, for IPCC AR5 one issue to be assessed by all three working groups is geoengineering, which hasn't been on the UNFCCC agenda yet, and I doubt it will be there anytime soon (i.e. within the next 5-8 years).

    In any case, I think your comment makes for a good hypothesis. Or do you have evidence to suggest it as fact?

    ReplyDelete
  16. @13- rjtklein

    "This isn't meant as sarcasm: how about having a look at the IPCC website if you want to find out more about the IPCC?"

    Oh, I've spent many, many hours on the IPCC website (not to mention those of its offshoots and "parents". But I'm curious: what is it about my comments here that lead you to believe that I had failed to "look at the IPCC website", as part of my self-education and due diligence?

    ReplyDelete
  17. -15-rjtklein

    It is a hypothesis with some pretty strong evidence in support for it .. for instance, compare the mentions of "carbon tax" to "carbon trading" in the AR4 (i.e., few vs. many).

    I am pretty sure we could come up with some objective measures of policy diversity and evaluate the trends with respect to those measures in IPCC WGIII reports. My operating hypothesis is a narrowing of that diversity, which as you say, would not be surprising, but would raise some questions about "neutral, but not prescriptive."

    ReplyDelete
  18. -16- hro001

    I must have misinterpreted your use of the word 'advocacy' and your saying that your due diligence only went back as far as AR4. Apologies.

    -17-

    I'm not an expert on Working Group III, but a quick scan of the AR4 Summary for Policymakers and Technical Summary doesn't suggest a significantly more frequent mention of trading as compared to taxes. But even if there were an imbalance in the underlying chapters, the reason doesn't have to be exclusively an inherent IPCC bias. There's long been an imbalance between the treatment by IPCC of mitigation and adaptation. To a large extent that reflected the interests of (most) governments, but also that there was (and still is) much less literature on adaptation than on mitigation. In any case, testing the hypothesis and explaining any findings would be an interesting exercise to do!

    As for the IPCC mantra, it's 'policy relevant, but not policy prescriptive'. The policy relevance is ensured, among other things, by having governments agree on the outlines of the reports.

    ReplyDelete
  19. -18-rjtklein

    Come now ... AR4 WGIII centers on implementation of the FCCC:

    http://www.ipcc.ch/pdf/assessment-report/ar4/wg3/ar4-wg3-chapter1.pdf

    ReplyDelete
  20. Hi Roger,

    I'm curious what your take is on the reception to the latest National Academy of Sciences climate change report, America's Climate Choices?

    According to several reports, the response on Capitol Hill seems to have been lukewarm at best (for example, see this round up: http://www.yaleclimatemediaforum.org/2011/07/academy-addresses-congress-questions/).

    You suggest in your column that a better approach is to make sure that the advisory process is more diverse and connected to formal decision processes. Depending on what you mean, the NAS panel seems to have embraced those principles in a couple of ways. One, unlike past reports, the committee included a diverse array of stakeholders, not just experts from academia, but also industry, NGOs, and even former politicians. And two, the committee was tasked by Congress to provide not just a snapshot of the state of the science, but to provide an overview of possible responses. The various panel reports go into a fair amount of detail about possible mitigation and adaptation strategies.

    Would you say the lukewarm response is an illustration of what you refer to at the end of your column as the policy maker's dilemma, or is there something that the Academy could have done to structure the study process differently to produce more impact?

    ReplyDelete
  21. -19- Roger

    "Come now ... AR4 WGIII centers on implementation of the FCCC:"

    I'm not arguing that at all. You said that emission trading was discussed more than carbon taxes. That's something that wasn't apparent to me from screening the Summary for Policymakers and the Technical Summary.

    ReplyDelete
  22. -21-rjtklein

    From a quick search of the WGII TS:

    "carbon tax" - 3 mentions
    "trading" -- 23 mentions
    "kyoto" - 29 mentions

    ReplyDelete
  23. -22- Roger

    Assuming you mean WGIII, I get the same numbers as you for 'trading' and 'kyoto'. The word 'tax' appears 35 times, many of which imply 'carbon'.

    The Kyoto Protocol doesn't at all exclude the possibility of countries introducing a carbon tax.

    Section 13 in the Technical Summary is explicitly about policies, instruments and co-operative agreements. It says:

    'The literature continues to reflect that a wide variety of
    national policies and measures are available to governments
    to limit or reduce GHG emissions. These include: regulations
    and standards, taxes and charges, tradable permits, voluntary
    agreements, phasing out subsidies and providing financial
    incentives, research and development and information
    instruments. Other policies, such as those affecting trade,
    foreign direct investments and social development goals can
    also affect GHG emissions.'

    The text that follows gives equal significance (in terms of word count) to each of these options, and does not recommend any one of them in particular.

    I think you'd like the final sentence:

    'There is a broad consensus in the literature that a successful
    agreement will have to be environmentally effective, cost-
    effective, incorporate distributional considerations and equity,
    and be institutionally feasible.'

    ReplyDelete
  24. -23-rjtklein

    You've picked a difficult position to defend;-) Let's dig a bit deeper ... Chapter 13 of WGIII - "Policies, instruments and co-operative arrangements" and as its title would suggest, is where you find a discussion of policy options.

    Some word counts:

    carbon (or emission or emissions) tax - 19

    emissions (or emission) trading - 90
    Kyoto - 89

    The IPCC WGIII has focused its attentions on implementation of Kyoto rather than other possible policy architectures. Implementation of Kyoto around the world has mostly (I'd say almost exclusively) focused on the establishment of emissions trading mechanisms, such as the ETS. The IPCC, rather than giving anything like "equal significance" to various policy options, has focused on a narrow subset of policy options.

    Such a focus in WGIII may or may not be desirable and may reflect factors external to the IPCC (such as where the academic literature has focused), but it seems unavoidably to be the case.

    ReplyDelete
  25. -24- Roger

    You subtly move away from the Technical Summary (without conceding that its relevant section 13 does give equal significance to each option) to the underlying chapter.

    You are right about the underlying chapter, and if this were a research projects (perhaps it should be!) we should interview the authors as to the reasons for the imbalance you find there. I could think of several, but it would be mere speculation coming from me.

    In any case, these kind of discussions show that we share a strong interest not only in science-policy interactions but also in collecting the evidence!

    ReplyDelete
  26. -25-rjtklein

    Thanks ... no, I don't concede the point about the TS, but I am happy to assert nolo contendre as it is not germane to my point, which is about the IPCC WGIII. You are right that the difference between the Chapter and TS is of note (and certainly not the only case).

    Look for a paper that quantitatively assesses the findings within the IPCC forthcoming very soon.

    ReplyDelete