19 July 2011

Sarewitz on NSF Broader Impacts

Dan Sarewitz has a column in Nature this week critical of NSF's well-meaning but ill-suited effort to modify its "broader impact criterion" -- the so-called "Criterion 2" --for the evaluation of specific project proposals.  Here is an excerpt (PDF):
Last month, the board published a revised criterion, and scientists had until this week to provide comments to the NSF before the final version is issued. But Criterion 2.1, as it might be called, is just as confusing and counterproductive as its predecessor.

At the heart of the new approach is “a broad set of important national goals”. Some address education, training and diversity; others highlight institutional factors (“partnerships between academia and industry”); yet others focus on the particular goals of “economic competitiveness” and “national security”. The new Criterion 2 would require that all proposals provide “a compelling description of how the project or the [principal investigator] will advance” one or more of the goals.

The nine goals seem at best arbitrary, and at worst an exercise in political triangulation. How else to explain the absence of such important aims as better energy technology, more effective environmental management, reinvigorated manufacturing, reduced vulnerability to natural and technological hazards, reversal of urban-infrastructure decay or improved performance of the research system? These are the sorts of goal that continue to justify public investments in fundamental research.

Yet, more troubling than the goals themselves is the problem of democratic legitimacy. In applying Criterion 2, peer-review panels will often need to choose between projects of equal intellectual merit that serve different national goals. Who gave such panels the authority to decide, for example, whether a claim to advance participation of minorities is more or less important than one to advance national security?
Sarewitz also makes this important point:
To convincingly assess how a particular research project might contribute to national goals could be more difficult than the proposed project itself.
Rather than asking PIs to do what is essentially impossible or at least far beyond their expertise, Sarewitz suggests that the locus of accountability needs to move higher up in the agency:
Motivating researchers to reflect on their role in society and their claim to public support is a worthy goal. But to do so in the brutal competition for grant money will yield not serious analysis, but hype, cynicism and hypocrisy. The NSF’s capacity to meet broad national goals is best pursued through strategic design and implementation of its programmes, and best assessed at the programme-performance level. Individual projects and scientists should be held accountable to specific programmatic goals, not vague national ones.
Ultimately, it is important to recognize that NSF is one of only three agencies with a legislated mandate to do "basic research" rather than agency-mission focused research (the two others being NASA and DOE's Office of Science). Asking NSF to become more mission oriented undercuts the overall purpose of the agency.

In my view (Sarewitz does not go this far) NSF should scrap the Criterion 2 at the project level and evaluate proposals on their scientific merit using peers with appropriate expertise. Setting the overall direction of NSF's research portfolios will inevitably be a political exercise, and that is a more appropriate location for efforts to better connect NSF with national needs.


  1. Typical rat-bag of requirements. The NSF should be encouraging the best possible science, not social engineering or economic planning.

  2. It's pretty darn stupid. Most institutions have developed a boilerplate paragraph of 'good deeds' that one can simply paste into a proposal. But on big grants, one often spends time trawling for stuff to pad out the 'broader impact' segment. For example, my school often includes promises to make high-end instrumentation available to local tribal colleges (we're short on historically African American institutions in Nebraska). Problem is, there really isn't anyone at the colleges we name who has the time or the expertise to make use of the instrumentation.

    If we really cared about tribal colleges, we'd make sure they were teaching really good freshman chemistry/physics/biology/calculus courses, not giving them theoretical 'access' to multimillion dollar instrumentation that research scientists only learn to make full use of as postdocs.

  3. Alas, Sarewitz is picking on the wrong branch of government. The national goals list is almost word for word from the America COMPETES Reauthorization Act from last summer (below). In short, they are stuck.


    (a) Goals.--The Foundation shall apply a Broader Impacts Review
    Criterion to achieve the following goals:
    (1) Increased economic competitiveness of the United States.
    (2) Development of a globally competitive STEM workforce.
    (3) Increased participation of women and underrepresented
    minorities in STEM.
    (4) Increased partnerships between academia and industry.
    (5) Improved pre-K-12 STEM education and teacher
    (6) Improved undergraduate STEM education.
    (7) Increased public scientific literacy.
    (8) Increased national security.

    (b) Policy.--Not later than 6 months after the date of enactment of
    this Act, the Director shall develop and implement a policy for the
    Broader Impacts Review Criterion that--
    (1) provides for educating professional staff at the
    Foundation, merit review panels, and applicants for Foundation
    research grants on the policy developed under this subsection;
    (2) clarifies that the activities of grant recipients
    undertaken to satisfy the Broader Impacts Review Criterion
    (A) to the extent practicable employ proven
    strategies and models and draw on existing programs and
    activities; and
    (B) when novel approaches are justified, build on
    the most current research results;
    (3) allows for some portion of funds allocated to broader
    impacts under a research grant to be used for assessment and
    evaluation of the broader impacts activity;
    (4) encourages institutions of higher education and other
    nonprofit education or research organizations to develop and
    provide, either as individual institutions or in partnerships
    thereof, appropriate training and programs to assist Foundation-
    funded principal investigators at their institutions in
    achieving the goals of the Broader Impacts Review Criterion as
    described in subsection (a); and
    (5) requires principal investigators applying for Foundation
    research grants to provide evidence of institutional support for
    the portion of the investigator's proposal designed to satisfy
    the Broader Impacts Review Criterion, including evidence of
    relevant training, programs, and other institutional resources
    available to the investigator from either their home institution
    or organization or another institution or organization with
    relevant expertise.

  4. Jean Goodwin is exactly right about the connection between the language of the proposed new criteria and the America COMPETES Reauthorization Act of 2010 -- but there's a bit more to the story.

    The language from the COMPETES Act comes from ... wait for it ... NSF!

    If you want to see for yourself, visit the CAPR Digital Repository here: http://csid-capr.unt.edu/fedora/repository/capr:15/-/Report%20in%20Response%20to%20America%20COMPETES%20Act:%20SEC.%207022.

    If you'd like a little analysis to go with that, check out my post on Science Progress here: http://www.scienceprogress.org/2010/09/accountable-science/.

    So, if they are stuck, they stuck themselves. However, as I argue in the Science Progress piece, if you take a look at the Report Language, you'll see that NSF is not stuck, after all. Not yet, anyway!

  5. Thanks for the references, Britt! This is a great example of the law of karma at work: you say it, you live it. Maybe NSF in the report you link thought it was *expanding* the goals list of COMPETES Act #1 by incorporating the economic and security bullet points?

    I wish I shared your optimism about wiggle room on the goals list. The old maxim from law--expressio unius, exclusio alterius/say one thing, exclude the others--why don't you think it will hold, here?

    Well, to look for a bright side, the push in the second part of the Act for *institutional* support for BI programs seems a good one. At my university, we're developing a program that links PIs with ongoing campus initiatives, especially on educational projects; see http://www.spisu.iastate.edu/ . This increases the effectiveness of the PI's efforts, since he/she can collaborate with relevant experts and adopt good practices. And at the same time it reduces the scope of the PI's efforts, since he/she just needs to provide the science expertise, not build a BI program from the ground up. It's a promising model, especially for a land grant institution already deeply involved in the community.

  6. What is the difference between the case of WMDs

    The construction of King Khalid Military City(The second largest military installation in the world after Fort Hood Texas) was authorized by Gerald Ford. Sometimes in geopolitics the 'policy decision' is made decades before a 'casus belli' presents itself.