Category Archives: Climate Proposal Review Process

How the NSF allocates billions of federal dollars to top universities by Lee Drutman

Figure 1 from the  Drutman article

There is an informative analysis of NSF funding in the article

How the NSF allocates billions of federal dollars to top universities by Lee Drutman

The article reads in part [highlight added]

As another college year begins, tens of thousands of academics will once again be scrambling to submit proposals to the National Science Foundation, hoping to secure government funding for their research. Each year, the National Science Foundation (NSF) bestows more than $7 billion worth of federal funding on about 12,000 research proposals, chosen out of about 45,000 submissions.

Thanks to the power of open data, we can now see how representation on NSF federal advisory committees connects to which universities get the most funding. (Federal advisory committee membership data is a feature of Influence Explorer.)

Our analysis finds a clear correlation between the universities with the most employees serving on the NSF advisory committees and the universities that receive the most federal money. Overall about 75% of NSF funding goes to academic institutions.

Even when controlling for other factors, we find that for each additional employee a university has serving on an NSF advisory committee that university can expect to see an additional $125,000 to $138,000 in NSF funding.

Although the 144 NSF advisory committees do not make funding decisions directly, they do “review and provide advice on program management, overall program balance, and other aspects of program performance,” according to the NSF.

At a big picture view, looking at the data on NSF grant awards and NSF advisory committee representation reinforces just how much of the money and representation is concentrated in a limited number of major universities.

Twenty percent of top research universities got 61.6% of the NSF funding going to top research universities between 2008 and 2011. These universities also had 47.9% of the representatives on NSF advisory committees who came from top research universities during the same period. The next 20% of universities got 21.9% of the funding, and had 25.7% of the representatives. The bottom 20% research universities had just 1.0% of the funding and have 2.4% of the representatives.

Just 23 universities account for more than half of the funding awarded by the NSF top to research universities. See Table 1 .

The University of California tops the list by far, because we combined all University of California campuses (due to data issues, see our data and methodology section), followed by Cal Tech, the University of Illinois, Michigan and Cornell. Interestingly, of the traditional top three universities (Harvard, Princeton and Yale), only Harvard shows up on the above list, at No. 22.

For complete data on 171 major research universities, click here. (The 171 universities come from the US News and World Report list of 200 major research universities. We selected only universities that had some interaction with the NSF between 2008 and 2011).

More representatives on advisory committees, more funding

Figure 1 plots the average NSF funding level for the university from 2008-2011, and the average number of representatives serving on NSF committees during this same period.

The correlation is clear. The more university-affiliated individuals serve on NSF advisory committees, the more NSF funding the university gets. Mostly, big state schools, with a few Ivy League schools in the mix, dominate the higher echelons of funding and representation. Interestingly, both Cal Tech and M.I.T., two of the pre-eminent research institutions in the country, get substantial NSF funding with limited representation. (Note: The University of California is left off this chart since it is a far outlier on both average funding ($361 million) and average representation (638.5 members). Because the quality of our data prevents us from breaking down the University of California by campus, we largely omit it from our analysis.)

A second scatterplot (Figure 2) examines the relationship between the number of committees and the funding levels. Here the data take on a slightly different relationship. With the exception of a few outliers, there is a changing relationship between the diversity of committees and the NSF funding levels.  It is more exponential than linear. Having representation on just a few committees doesn’t consistently correlate with higher funding, but having representation on a lot of committees is strongly correlated with higher funding.

Do more representatives help universities secure more funding?

The NSF “strives to conduct a fair, competitive, transparent, merit-review process for the selection of projects,” based on intellectual merit and broader impacts. Each year, the NSF produces an annual report on the merit review process. To make funding decisions, the NSF relies on tens of thousands of expert reviewers, though program officers make the final decisions.

Advisory committees oversee the general direction of the NSF program areas, including identifying “disciplinary needs and areas of opportunities.” As for who gets on these committees, the NSF explains that: “Many factors are weighed when formulating Committee membership, including the primary factors of expertise and qualifications, as well as other factors including diversity of institutions, regions, and groups underrepresented in science, technology, engineering, and mathematics.”

An example of such a committee is the Proposal Review Panel for Information and Intelligent Systems. Following the hyperlink provided would take you to a list of committee members in Influence Explorer, most of whom have university affiliations.

Showing that more representatives help universities get more funding than they would otherwise have received is difficult. There is a very good and reasonable explanation for the patterns we observe in the two above scatter plots: The NSF tries to get the most knowledgeable experts and accomplished academics to serve on its committees. Not surprisingly, the universities that attract the most NSF money are also likely to be home to many accomplished experts, since they are all leading research universities.

However, there are a few ways in which representatives could help their own universities to improve their chances. One possibility is that if a department has a representative on an NSF committee, that representative will be able to pass along funding opportunities and advice on navigating on the decision-making process of the committee to others in the university, thus strengthening others’ chances. Insiders can help others to better understand what a review committee might be looking for.

Another possibility is that in directing the general funding strategies of NSF program areas, advisory committees might see what their universities are doing as particularly valuable. Or more benignly, they might be more aware of the cutting-edge research within their universities just because it is being done by colleagues they interact with on a regular basis.

One way to investigate the relationship is to do a regression analysis, which allows us to control for different factors simultaneously. For those of a more technical mind, the details are below. For those who want the quick takeaway, it goes like this: Controlling for previous NSF funding and university endowment, universities with more NSF advisory committee representatives get more NSF funding than those that don’t. Each additional representative translates into about an extra $125,000 to $138,000 in NSF funding, controlling for other factors. The number of representatives is more important than the number of committees with representatives. Lobbying expenditures make no difference.

The entire article with the tables is very worth reading.

Comments Off on How the NSF allocates billions of federal dollars to top universities by Lee Drutman

Filed under Climate Proposal Review Process

Are Some Climate Scientists And Program Managers At NSF Gaming the System With Respect To Providing Inadequately Assessed Multi-Decadal Predictions Of Changes In Climate Statistics To The Impacts Community?

Are climate scientists and program managers who fund them, who  misrepresent multi-decadal climate model predictions to the impacts communities as skillful projections,  “gaming the system” ?  Gaming the system is defined as

Gaming the system (or bending the rules, playing the system, abusing the system, milking the system or working the system) can be defined as “[using] the rules and procedures meant to protect a system in order, instead, to manipulate the system for [a] desired outcome”.

My answer is YES, unless these individuals can demonstrate skill at multi-decadal regional climate predictions which they fund. To my knowledge, however, this demonstration of skill has not been shown. Thus those individuals are either very naive or are deliberately gaming the system.

I am quite blunt in this post (and see also this post ) where I focus on the NSF, since the individuals I am referring to have been alerted to the failings with respect to providing multi-decadal predictions of changes in regional and local climate statistics to the impacts community.  To be clear, my criticism is not with all of NSF, but only with those program managers who are funding multi-decadal impact studies based on predictions of changes in regional and local climate statistics.

Despite the identification of this issue and either refuting our findings, or accepting them and not continuing to fund, they have ignored this issue.  I see no other explanation for them ignoring our findings, except that they can use their claims of providing forecasts for the policymakers and stakeholders in order to continue the flow of dollars from the federal treasury into their programs.

At best this is short-sighted, and at worse it is dishonest.

As I discussed in my post

Short Circuiting The Scientific Process – A Serious Problem In The Climate Science Community

There has been a development over the last 10-15 years or so in the scientific peer-reviewed literature that is short circuiting the scientific method.

The scientific method involves developing a hypothesis and then seeking to refute it. If all attempts to discredit the hypothesis fails, we start to accept the proposed theory as being an accurate description of how the real world works.

A useful summary of the scientific method is given on the website they list six steps

  • Ask a Question
  • Do Background Research
  • Construct a Hypothesis
  • Test Your Hypothesis by Doing an Experiment
  • Analyze Your Data and Draw a Conclusion
  • Communicate Your Results

Unfortunately, in recent years papers have been published in the peer-reviewed literature that fail to follow these proper steps of scientific investigation. These papers are short circuiting the scientific method.

Many of these peer-reviewed papers are funded by the NSF.

As of today’s date, it is clear they are still ignoring addressing the issues that we have summarized in our peer-reviewed articles – Pielke and Wilby 2012 and Pielke et al 2012They are gaming the system in order to continue the high level of funding for impact studies that are based on the multi-decadal regional and local climate model predictions of changes in climate statistics.

While I support the NSF funding of the assessment of predictability; e.g. see below for an example from KNMI in the Netherlands

Guest Post Titled “Decadal Prediction Skill In A Multi-Model Ensemble” By Geert Jan van Oldenborgh, Francisco J. Doblas-Reyes, Bert Wouters, Wilco Hazeleger

Seminar Announcement – On The Reliability Of Climate Models: How Well Do They Describe Observed Trends? By Geert Jan van Oldenborgh Of KMNI

The Difference Between Prediction and Predictability – Recommendations For Research Funding Related to These Distinctly Different Concepts

the provision of multi-decadal regional and local climate model results for the coming decades and representing them as skillful projections is scientifically invalid. Program managers, principal investigators, policymakers and others who report the results as skillful and do not include the needed disclaimer that I presented in my post

Climate Science Malpractice – The Promotion Of Multi-Decadal Regional Climate Model Projections As Skillful

are certainly gaming the system (or are very naive).

Remarkably, as I have learned, there is no accountability and review of what the NSF program managers fund in terms of topic area, including no effective mechanism to contest if they are actually funding robust scientifically tested research.

They do not even need to retain copies of their e-mails to ascertain how they handle issues such as raised in our papers and in my weblog posts, but can delete them so they cannot be obtained under an FOIA request. I discussed this failing in my post

My Experiences With A Lack Of Proper Diligence And Bias In The NSF Review Process For Climate Proposals

where I concluded that

  • NSF does not retain a record of e-mail communications
  • NSF is cavalier in terms of the length of time proposals are under review.
  • NSF has decided to emphasize climate modeling and of funding multi-decadal climate predictions, at the expense of research which can be tested against real-world observations.
  • NSF penalizes scientists who criticize their performance.

With respect to e-mails (and this part of NSF accountability),  as was communicated to me by a lawyer at the NSF

On Fri, 15 Apr 2011, Jensen, Leslie A. wrote:

Dear Sir:

A proper search has been accomplished for all named individuals.  Email is  not a permanent record and meetings from several years ago would be deleted.  The Foundation’s email retention policy is repeated below:


Exchange Server: Most users have their mail delivered to their Exchange  Server mailbox. If you haven’t done anything special, that is where your  mail is delivered and stored. In Outlook your mailbox is the folder that  includes your name in the folder name (top folder in your Folder List). The  Exchange Servers are backed up to tape nightly, and the tapes are retained  for 14 days, then destroyed. Exchange Server has a feature that allows you  to recover deleted messages (even after the trash is emptied). That feature  is set to retain deleted messages for 5 days. When these features are combined, it means that 19 days after you delete a message and empty the  trash, nobody can recover it.

This lack of accountability with federally supported NSF should be of concern to everyone, regardless of your perspective on the climate science issue.

In my post

Climate Assessment Oligarchy – The IPCC

I wrote with respect to the IPCC

An oligarchy is a

“form of government in which all power is vested in a few persons or in a dominant class or clique; government by the few.”

This definition can certainly also be applied to  a number of program managers at the NSF, and other funders, of climate science research. Until and unless a new direction that is actually based on the scientific method is introduced, we will continue to see this abuse of their positions (and waste of funds) as stewards of funding scientific research.

source of image

Comments Off on Are Some Climate Scientists And Program Managers At NSF Gaming the System With Respect To Providing Inadequately Assessed Multi-Decadal Predictions Of Changes In Climate Statistics To The Impacts Community?

Filed under Climate Proposal Review Process, Climate Science Misconceptions

Follow Up On My Post “Funding Agency Bias – A Short Summary”

Don Bishop, in response to the post

Funding Agency Bias – A Short Summary

sent the informative e-mail below which I invited him to share with everyone. It is reproduced with his approval.

Date: Tue, 27 Mar 2012 11:45:13 -0400 (EDT) From: Don Bishop Subject: Agency funding bias

Hi Roger

While reading today’s post [that of March 27 2012] about bias, and how models do not give useful regional weather/climate predictions, I was reminded of Judy Curry’s presentation in Boulder on August 31, 2011 – an example of what is needed.

The workshop was sponsored by NOAA on water cycle science challenges, and Curry’s presentation showed how lower 48 droughts relate to PDO and AMO, what  was predicted for those ocean cycles, and therefore what the drought predictions were: the analogue for the next decade or so was the 1950s.

_ (

Her slides:

_ y_noaaWaterClimate.pdf_ (

Don Bishop

Comments Off on Follow Up On My Post “Funding Agency Bias – A Short Summary”

Filed under Climate Proposal Review Process

The NSF CREATIV Initiative On Interdisciplinary Research – Another Example Of Thinking Inside The Box

Several months ago, the NSF announced the CREATIV initiative in  a Dear Colleague letter.  Since we have proposed a broader, interdisciplinary  approach to the assessment of risks to key societal and environmental risks in our paper

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2011: Dealing  with complexity and extreme events using a bottom-up, resource-based  vulnerability perspective. AGU Monograph on Complexity and  Extreme Events in Geosciences, in press

I submitted an application to submit based on two issues:

1. The assessment of the skill at the top-down global climate model predictions of changes in climate statistics that are relevant to stakeholders in terms of the metrics they need


2. The quantification of the climate and other threats to these metrics with respect to selected key resources in water, food, energy, human health and ecosystem function.

My experience, that I have documented below, with the  program managers who handled this issue illustrates the problem with obtaining funding at the NSF in climate related studies unless you fit into their particular area of interest (i.e. there are “favored topics” despite what is announced in the Dear Colleague letter).

The CREATIV program itself is described as

CREATIV (Creative Research Awards for Transformative Interdisciplinary Ventures): a pilot grant mechanism under the Integrated NSF Support Promoting Interdisciplinary Research and Education (INSPIRE) initiative, to support bold interdisciplinary projects in all NSF-supported areas of science, engineering, and education research.

The CREATIV program has these claimed goals

  • Create new interdisciplinary opportunities that are not perceived to exist presently.
  • Attract unusually creative high-risk / high-reward interdisciplinary proposals.
  • Provide substantial funding, not limited to the exploratory stage of the pursuit of novel ideas.
  • Designate no favored topics; be open to all NSF-supported areas of science, engineering, and education research.

In terms of “no favored topics” they write

No. There are no  favored topics. In terms of review  criteria, unusual promise for societal benefit can contribute to the broader  impacts of a proposal.

In the following, I have reproduced the e-mails involved in my interaction with the program managers at the NSF.  I have redacted their names, since these individuals just illustrate a culture at the NSF in the area of climate science that is clearly biased to perpetuating a particular viewpoint on the climate issue.

My First E-mail

Dear sir/madam

Well over a month ago, I submitted the short write up at the NSF website (using your form – on submitting a CREATIV proposal, but have not heard anything regarding my submission.

Please let me know who I should contact regarding its status. CREATIV is described at

and was announced by Professor (NSF Director) Suresh, who I have copied to on his MIT e-mail address.

I submitted my information as requested in the text

“Potential proposers are encouraged to begin the process by submitting the CREATIV Inquiry Data Form, as explained on the FAQ page. Before writing and submitting a CREATIV proposal, it is the principal investigator’s responsibility to obtain written authorization to submit a CREATIV proposal by NSF program directors from at least two intellectually distinct divisions or programs.”

I completed this step [from]

“As a first step, you are encouraged to fill out and submit the CREATIV Inquiry Data Form, here. Before submitting the form, you must identify at least one appropriate NSF Program Director to consider your inquiry.”

We listed a number of NSF Program Directors to assess and based our request on our interdisciplinary paper

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2012: Dealing with complexity and extreme events using a bottom-up, resource-based vulnerability perspective. AGU Monograph on Complexity and Extreme Events in Geosciences, in press.

However, so far, we have had no feedback from the NSF on our Inquiry submission.


Roger A. Pielke Sr.

My Response: Here is the first response which I received only after I copied the Director of the NSF.

Dear Dr. Pielke,

I am an NSF program officer.  I was not among the program officers that you listed on your CREATIV inquiry, but I am among those to whom your inquiry was routed.

It seems that perhaps the following NSF grants are already funding aspects of what you are proposing:

A. RESIN Grants from ENG/EFRI (RESIN = Resilient and Sustainable Infrastructure).  One example:

EFRI-RESIN: Assessing and Managing Cascading Failure Vulnerabilities of Complex, Interdependent, Interactive, Adaptive Human-based Infrastructure Systems

B. WSC Grants (WSC = Water Sustainability and Climate).  Some examples:

WSC-Category 2: Extreme events impacts on water quality in the Great Lakes: Prediction and management of nutrient loading in a changing climate

Collaborative Research, WSC-Category 2: Regional Climate Variability and Patterns of Urban Development – Impacts on the Urban Water Cycle and Nutrient Export

C. RCN-SEES Grants (RCN-SEES = Research Coordination Networks – Science, Engineering, and Education for Sustainability).  Some examples:

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

So, I do not feel that the CREATIV path is a fit for your concept.  One path forward might be for you to put together an RCN-SEES proposal that would be built around your approach but would draw upon related ongoing NSF-funded research and researchers such as are described at the links above.  The next RCN-SEES deadline is already posted (Feb. 4, 2013).  Here is a link:

Another option might be to pursue CaMRA, as described in NSF’s posted FY 2013 Budget Request: “Creating a More Disaster-Resilient America (CaMRA) aims to catalyze basic research and education efforts in hazard-related science, engineering, risk assessment and decision making in order to improve forecasting and prediction of natural and technological hazards, mitigate their effects, and prepare communities to respond to, and recover from disasters.”

Best wishes,


My Reply


Thank you for your detailed and thoughtful reply.

These programs certainly would fit with out bottom-up focus, but a deadline of February 2013 for the next RCN-SEES deadline (Feb. 4, 2013) for just submitting, means it would be 1 1/2 years or more from now before we would have any funding, even if we were successful in a proposal.

Since our research; as presented in our papers

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2012: Dealing with complexity and extreme events using a bottom-up, resource-based vulnerability perspective. AGU Monograph on Complexity and Extreme Events in Geosciences, in press.

Pielke Sr., R.A., and R.L. Wilby, 2012: Regional climate downscaling – what’s the point? Eos Forum, 93, No. 5, 52-53, doi:10.1029/2012EO050008.

shows that the current approach of regional and local downscaling of multi-decadal global model climate predictions for use to determine vulnerabilities to key societal and environmental resources is seriously flawed, we feel it is important to examine the value of the bottom up approach. The IPCC type scenarios are, at best, a subset of what climate threats that will be faced in the future.

The CREATIV option seems to be the only option to use in the near term. Please advise, however, if the programs such as

EFRI-RESIN: Assessing and Managing Cascading Failure Vulnerabilities of Complex, Interdependent, Interactive, Adaptive Human-based Infrastructure Systems

WSC Grants (WSC = Water Sustainability and Climate).

Collaborative Research, WSC-Category 2: Regional Climate Variability and Patterns of Urban Development – Impacts on the Urban Water Cycle and Nutrient Export

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

have any closer deadlines to submit [I assume the last two have a Feb 2013 date for new submissions).



The Response:

 Hello Roger,

My colleague YYYY has kindly provided information on WSC below.

There is some chance that EFRI-RESIN may have another solicitation, but that is not yet decided, and, at any rate, the deadline would be after the next RCN-SEES deadline of February 4, 2013.

In my message below (with relevant links), the following two are already existing NSF RCN-SEES grants that I was suggesting may have a relationship to your concept (they are not funding opportunities separate from RCN-SEES with the deadline of February 4, 2013):

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

The Response from the other NSF Program Officer:


The Water Sustainability and Climate (WSC) competition is likely to have its next deadline in around 18 months, although this is uncertain and the specific requirements for proposals may change. The current solicitation for large awards is funding research that focuses intensely on one watershed or that compares water sustainability issues in two or more watersheds. The solicitation requires that proposals integrate the work of experts from fields in engineering, geosciences, biosciences, and social sciences. Some of what you want to do may well fit with WSC.

Best wishes, YYYY

My Response


Thank you for your feedback. What I am interested in is not large funding, but about 1750K to 200K per year for three years to further examine:

i) shortcomings of the top-down (IPCC; CCSP) (outcome) vulnerability approach and,

ii) provide examples of the more holistic, inclusive assessment of vulnerabilities in our bottom-up (contextual) approach.

Since I am leading, as Editor-in-Chief, a 5 volume set of books which introduce this perspective (and provides examples), I would be able to leverage from the work that we are producing in this volume and to work with a number of those colleagues to assist in building on it. One issue with the set of 5 books is that there remains considerable existing emphasis on the narrower top-down, global climate model dominated view, and a focused study on the bottom-up approach is needed.

My Editors are Faisal Hossain on water resources (Tennessee Technological University); Jimmy Adegoke and Caradee Wright on human health (CIRS; South Africa); Tim Seasteadt and Katie Suding on ecosystem function (University of Colorado/ UC Berkeley); Dev Niyogi on food (Purdue); and George Kallos on energy (University of Athens, Greece).

My proposal is very straightforward. It is to determine the major threats to a selected examples of local and regional water, food, energy, human health, and ecosystem function resources to short term weather events (e.g. tornadoes), multi-weekly and seasonal long events (droughts) and multi-year and decadal variability and change (e.g. an increase or decrease in the 30 year warm season maximum temperatures), and also from other social and environmental issues. After these threats are identified for each resource, then the relative risks from weather and climate issues can be compared with other risks in order to adopt optimal preferred mitigation/adaptation strategies.

The questions to be asked to a selected group of stakeholders (chosen by working with those who worked with me as Editors) are:

1. Why is this resource important? How is it used? To what stakeholders is    it valuable?

2. What are the key environmental and social variables that influence this    resource?

3. What is the sensitivity of this resource to changes in each of these    key variables? (This includes, but is not limited to, the sensitivity    of the resource to climate variations and change on short (days);    medium (seasons) and long (multi-decadal) time scales).

4. What changes (thresholds) in these key variables would have to    occur to result in a negative (or positive) outcome for this resource?

5. What are the best estimates of the probabilities for these changes to    occur? What tools are available to quantify the effect of these    changes? Can these estimates be skillfully predicted?

6. What actions (adaptation/mitigation) can be undertaken in order to    minimize or eliminate the negative consequences of these changes (or to    optimize a positive response)?

7. What are specific recommendations for policymakers and other    stakeholders?

I hope you can direct me to where I can seek funding for this effort prior to 2013.



The Response 


To the extent that your project is descriptive and essentially a literature review, the project is probably not appropriate for NSF whose mission is to support basic research.

If elements of your project are likely to produce a theoretical or methodological advance, then the project is probably appropriate for the DRMS Program. The next target date is August 18 with funding decisions in November and December.

Best wishes,


My Reply


The approach is quite a bit more than a literature survey, as stakeholders would need to be interviewed and data analyzed. It is a more robust approach the current top-down method, as we concluded in our AGU book chapter and the EOS article. The top-down approach is currently receiving quite large amounts of NSF funding, yet the basic science robustness of this approach has not been adequately vetted.  This would be part of our research.

I do see under list of activities for the Decision, Risk and Management Sciences (DRMS) -

that you fund small grants that are high-risk and of a potentially transformative nature (EArly-Concept Grants for Exploratory Research – EAGER).

Our approach would be to test if the current top-down global climate model multi-decadal prediction approach is fatally flawed as a major driver of impact studies(as we have found so far in our studies), and that funds are being wasted on this methodology. This would be a transformative finding if our finding is confirmed.

To my knowledge, there is no coherent NSF program that is looking into this issue.

Best Regards


Their Reply


The DRMS proposal officers have concluded that your proposed project does not seem to be sufficiently risky and transformational to warrant DRMS encouraging you to submit an EAGER proposal. The notion that one can usefully look at the incremental threat to a sector from any particular hazard is not a great conceptual leap forward. There is no reason your proposed project should not undergo NSF’s peer review procedures.

Best wishes,


My Reply


Where in the NSF (or elsewhere for that matter) is the assessment of the skill of the IPCC-type models in the prediction of changes in multi-decadal regional climate statistics being completed? It is these model results that are being given to the impacts communities. If our conclusions, as summarized in the peer reviewed articles I sent to you, are correct, NSF is wasting a lot of money. It would seem this is more than an incremental issue. There are two parts to what I have proposed, and this assessment of climate prediction skill is the first part. I am familiar, and support the assessment of the decadal predictability initiative of the NSF (as an initial condition), but this is distinct from the use of the multi-decadal climate change predictions. Best Regards Roger

My Second Reply since I was not getting a response


Do you plan to reply to my request for information as to which NSF funded projects are assessing the skill of multi-decadal regional climate predictions?

The regional climate model predictions for the coming decades that are being provided to the impacts communities is not basic science unless their predictive skill can be determined.

To my knowledge, however [and as we summarized in the Pielke and Wilby, 2012 EOS article], they have not been shown to have skill even in a hindcast mode when compared with observed variations and long term changes in climate statistics.

This assessment is very much needed (unless you or others provide information that refutes my finding) and is one of the two pillars of my proposed study. This, as I have written, is hardly an incremental study, but underpins a large amount of spending by the NSF.

I also plan to post regarding this issue on my weblog in the coming days, but would like your feedback first on my question.

Best Regards

The Response:


The appropriate responder to your question is in the Geological Sciences Directorate, not the Social, Behavioral and Economic Sciences Directorate. I do not know if NSF is funding projects that are assessing the skill of multi-decadal regional climate predictions.

Best wishes,


My Reply


Thank you for the quick reply.

Please let me know who I should contact in the Geological Sciences  Directorate.

In terms of the Social, Behavioral and Economic Directorate, however, I am under the understanding that climate impact studies on society are funded  which use (and have their fundamental basis on) the regional climate model  projections. If my understanding is incorrect on this, please let me know.

Best Regards


Their Reply


One good way to find out what NSF is funding is to check out abstracts of  awards listed on the NSF website ( You can search by keyword,  program, etc. The website also describes the different programs in the  atmospheric sciences division of the geosciences directorate.

Best wishes,


My Reply


We have already done that and there are no grants that we found that assess the model skill as I have outlined in our papers. They are quite well aware of the issues I have raised on this subject in the Geosciences Directorate and have chosen to ignore them. The only exception is their new focus on decadal predictability, which, however, is still distinct from the multi-decadal impact studies based on the IPCC type projections.

My point of seeking funding outside is

i) that programs that examine societal and environmental vulnerability are using the results from the multi-decadal global climate model predictions as an essential part of their studies. These results are then being used (misused in my view) by the policy and political communities.


ii) The Geosciences Directorate has chosen to arbitrarily accept the model predictions as applied to impact studies as robust. They have ignored, so far, attempts to get them to properly assess this issue.

My bottom-line conclusion is that there is really no basic science aspect to these NSF funded impact studies [which appear across the NSF funding spectrum], yet NSF continues to fund them.

Best Regards


Their Reply       None

source of image

Comments Off on The NSF CREATIV Initiative On Interdisciplinary Research – Another Example Of Thinking Inside The Box

Filed under Climate Proposal Review Process

NSB/NSF Seeks Input on Proposed Merit Review Criteria Revision and Principles

The National Science Board has sent out a notice requesting input on the NSF review process. Their request is reproduced in this post. One glaring issue that is missing is accountability. I discussed this subject in my posts

My Experiences With A Lack Of Proper Diligence And Bias In The NSF Review Process For Climate Proposals

Is The NSF Funding Untestable Climate Predictions – My Comments On A $6 Million Grant To Fund A Center For Robust Decision–Making On Climate And Energy Policy”

The National Science Foundation Funds Multi-Decadal Climate Predictions Without An Ability To Verify Their Skill

NSF Decision On Our Request For Reconsideration Of A Rejected NSF Proposal On The Role Of Land Use Change In The Climate System

Is The NSF Funding Process Working Correctly?

I have made the following recommendations:

  • Guarantee that the review process be completed within 6 months [my most recent land use and climate proposal was not even sent out for review until 10 months after its receipt!)
  • Retain all e-mail communications indefinitely (NSF staff can routinely delete e-mails, such that there is no record to check their accountability)
  • Require external independent assessments, by a subset of scientists who are outside of the NSF, of the reviews and manager decisions, including names of referees. This review should be on all accepted and rejected proposals ( as documented in the NSF letter at the end of this post, since they were so late sending out for review, they simply relied on referees of an earlier (rejected) proposal; this is laziness at best).

The National Science Board request follows. I will be submitting my comments, based on the above text, and urge colleagues who read my weblog to do likewise.

NSB/NSF Seeks Input on Proposed Merit Review Criteria Revision and Principles

National Science Board
June 14, 2011

Over the past year, the National Science Board (NSB) has been conducting a review of the National Science Foundation’s merit review criteria (Intellectual Merit and Broader Impacts). At the Board’s May 2011 meeting, the NSB Task Force on Merit Review proposed a revision of the two merit review criteria, clarifying their intent and how they are to be used in the review process. In addition, the Task Force identified a set of important underlying principles upon which the merit review criteria should be based. We now seek your input on the proposed revision and principles.

The Task Force looked at several sources of data for information about how the criteria are being interpreted and used by the NSF community, including an analysis of over 190 reports from Committees of Visitors. The Task Force also reached out to a wide range of stakeholders, both inside and outside of NSF, to understand their perspectives on the current criteria. Members of NSF’s senior leadership and representatives of a small set of diverse institutions were interviewed; surveys about the criteria were administered to NSF’s program officers, division directors, and advisory committee members and to a sample of 8,000 of NSF’s Principal Investigators (PIs) and reviewers; and the NSF community at large was invited to provide comments and suggestions for improvements through the NSF web site ( The stakeholder responses were very robust—all told, the Task Force considered input from over 5,100 individuals.

One of the most striking observations that emerged from the data analyses was the consistency of the results, regardless of the perspective. All of the stakeholder groups identified similar issues, and often offered similar suggestions for improvements. It became clear that the two review criteria of Intellectual Merit and Broader Impacts are in fact the right criteria for evaluating NSF proposals, but that revisions are needed to clarify the intent of the criteria, and to highlight the connection to NSF’s core principles.

The two draft revised criteria, and the principles upon which they are based, are below. Comments are being collected through July 14—we invite you to send comments to It is expected that NSF will develop specific guidance for PIs, reviewers, and NSF staff on the use of these criteria after the drafts are finalized. Your comments will help inform development of that guidance, and other supporting documents such as FAQs.

The Foundation is the primary Federal agency supporting research at the frontiers of knowledge, across all fields of science and engineering (S&E) and at all levels of S&E education. Its mission, vision and goals are designed to maintain and strengthen the vitality of the U.S. science and engineering enterprise and to ensure that Americans benefit fully from the products of the science, engineering and education activities that NSF supports. The merit review process is at the heart of NSF’s mission, and the merit review criteria form the critical base for that process.

We do hope that you will share your thoughts with us. Thank you for your participation.

Ray M. Bowen
Chairman, National Science Board
Subra Suresh
Director, National Science Foundation


Merit Review Principles and Criteria
The identification and description of the merit review criteria are firmly grounded in the following principles:

  1. All NSF projects should be of the highest intellectual merit with the potential to advance the frontiers of knowledge.
  2. Collectively, NSF projects should help to advance a broad set of important national goals, including:
    • Increased economic competitiveness of the United States.
    • Development of a globally competitive STEM workforce.
    • Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
    • Increased partnerships between academia and industry.
    • Improved pre-K–12 STEM education and teacher development.
    • Improved undergraduate STEM education.
    • Increased public scientific literacy and public engagement with science and technology.
    • Increased national security.
    • Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships.
  3. Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research. All are valuable approaches for advancing important national goals.
  4. Ongoing application of these criteria should be subject to appropriate assessment developed using reasonable metrics over a period of time.

Intellectual merit of the proposed activity

The goal of this review criterion is to assess the degree to which the proposed activities will advance the frontiers of knowledge. Elements to consider in the review are:

  1. What role does the proposed activity play in advancing knowledge and understanding within its own field or across different fields?
  2. To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
  3. How well conceived and organized is the proposed activity?
  4. How well qualified is the individual or team to conduct the proposed research?
  5. Is there sufficient access to resources?

Broader impacts of the proposed activity

The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national goal(s). Elements to consider in the review are:

  1. Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling description of how the project or the PI will advance that goal(s)?
  2. Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
  3. Is the rationale for choosing the approach well-justified? Have any innovations been incorporated?
  4. How well qualified is the individual, team, or institution to carry out the proposed broader impacts activities?
  5. Are there adequate resources available to the PI or institution to carry out the proposed activities?

Comments Off on NSB/NSF Seeks Input on Proposed Merit Review Criteria Revision and Principles

Filed under Climate Proposal Review Process

Predicting Wars Based On Climate Change – Really?

There is a remarkable recently NSF funded study that claims to be able predict the liklihood of wars based in part on multi-decadal climate change. Readers can assess from their own perspectives if this is appropriate as a scientifically testable study. 

In my view, a significant portion of NSF funding is going to projects that involve predictions decades into the future, and this is just one example.

 These predictions, of course, cannot be verified during the lifetime of the project (and even for years afterwards), so it is not clear to me why such research is being funded by the NSF. 

Here is the study with an extract from the abstract.  

Collaborative Research: Climate Change and Variability and Armed Conflicts in Africa South of the Sahara

“The number of armed conflicts has declined after the end of the Cold War. There is also a long-term trend towards less severe armed conflicts, though climate change threatens to reverse this favorable trend……Using a predictive model of the coupled natural (climate) and social (violence) systems, with feedback loops and mediating socio-political-economic variables, the PIs will measure the impact of adverse climate change and/or changes in climate variability on the rate of armed conflict, determine which mediating factors influence the rate of this impact, and project the violence outcomes on the basis of different climate change/variability scenarios.”

Comments Off on Predicting Wars Based On Climate Change – Really?

Filed under Climate Proposal Review Process, Climate Science Misconceptions

NSF Decision On Our Request For Reconsideration Of A Rejected NSF Proposal On The Role Of Land Use Change In The Climate System

On May 18 2010 I posted on a proposal to NSF that was highly rated each time it was submitted, but was rejected each of the three times it was submitted after further revisions were made. 

The May post is

Is The NSF Funding Process Working Correctly?

The NSF mission reads

The National Science Foundation Act of 1950 (Public Law 81-507) set forth NSF’s mission and purpose:

To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense….

The Act authorized and directed NSF to initiate and support:

  • basic scientific research and research fundamental to the engineering process,
  • programs to strengthen scientific and engineering research potential,
  •  science and engineering education programs at all levels and in all the various fields of science and engineering,
  • programs that provide a source of information for policy formulation,
  • and other activities to promote these ends.

Over the years, NSF’s statutory authority has been modified in a number of significant ways. In 1968, authority to support applied research was added to the Organic Act. In 1980, The Science and Engineering Equal Opportunities Act gave NSF standing authority to support activities to improve the participation of women and minorities in science and engineering. Another major change occurred in 1986, when engineering was accorded equal status with science in the Organic Act.

NSF has always dedicated itself to providing the leadership and vision needed to keep the words and ideas embedded in its mission statement fresh and up-to-date. Even in today’s rapidly changing environment, NSF’s core purpose resonates clearly in everything it does: promoting achievement and progress in science and engineering and enhancing the potential for research and education to contribute to the Nation. While NSF’s vision of the future and the mechanisms it uses to carry out its charges have evolved significantly over the last four decades, its ultimate mission remains the same.

The title and Project Summary of our rejected proposal is

“Collaborative Research: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes since European Settlement”

with the Project Summary

“The Earth’s weather and climate is strongly influenced by the properties of the underlying surface. Much of the solar energy that drives the atmosphere first interacts with the land or sea surface. Over land regions this interaction is modulated by surface characteristics such as albedo, aerodynamic roughness length, leaf area index (LAI), etc. As these characteristics change, either from anthropogenic or natural land-cover disturbances, the amount of energy reaching the atmosphere from the land surface, and thus weather and climate, is expected to change. The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System (RAMS) coupled with the Simple Biosphere (SiB) model, SiB-RAMS, will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox (2008). The influence of the land-cover changes on temperature and precipitation will be examined and compared with that expected from CO2-induced climate change (IPCC 2007). The seasonality of the changes in precipitation and temperature due to land-cover change will be explored. Also, the relative importance of each land-cover biophysical parameter to the total simulated change in temperature and precipitation will be assessed.”

We have presented the letter from the Deputy Assistant Director regarding our request for reconsideration.

This letter is quite informative as it is a cursory, pro forma response without any detail. What it confirms is that program managers have considerable latitude in decision-making and can eliminate well reviewed projects if they differ from their priorities. The program managers decide what is “basic scientific research and research fundamental to the engineering process” rather than relying on the reviewers to determine this [of course, they can also select known biased reviewers if they want to reject a proposal].

Since the level of ratings of our proposal were high, the reason for the rejection is based on the program managers concluding that the role of land use change in the climate system is not a high research priority.  Also, despite the NSF requirement listed their mission statement “to support activities to improve the participation of women and minorities in science and engineering”,  the fact that woman (Dr. Lixin Lu) was the PI on the project was not discussed in the reconsideration.

My recommendation to improve the process, which I presented in my May post is

  • present ALL proposal abstracts, anonymous reviews of both accepted and rejected proposals and program managers decision letters (or e-mails) on-line for public access
  • present the date of submission and final acceptance (or rejection) of the proposal.
  • I also recommend they make easily available the list of all of the reviewers used during the year within each NSF program office.

    NSF program managers have considerable ability to slant research that they fund with insufficent transparency of the review process. This has become quite a problem in the climate science area where, as one example, in recent years they have elected to fund climate predictions decades into the future (e.g. see which was funded in part by the NSF; I will discuss specific examples of such funded projects by the NSF in a future post).

    Comments Off on NSF Decision On Our Request For Reconsideration Of A Rejected NSF Proposal On The Role Of Land Use Change In The Climate System

    Filed under Climate Proposal Review Process