How the NSF allocates billions of federal dollars to top universities by Lee Drutman

Figure 1 from the  Drutman article

There is an informative analysis of NSF funding in the article

How the NSF allocates billions of federal dollars to top universities by Lee Drutman

The article reads in part [highlight added]

As another college year begins, tens of thousands of academics will once again be scrambling to submit proposals to the National Science Foundation, hoping to secure government funding for their research. Each year, the National Science Foundation (NSF) bestows more than $7 billion worth of federal funding on about 12,000 research proposals, chosen out of about 45,000 submissions.

Thanks to the power of open data, we can now see how representation on NSF federal advisory committees connects to which universities get the most funding. (Federal advisory committee membership data is a feature of Influence Explorer.)

Our analysis finds a clear correlation between the universities with the most employees serving on the NSF advisory committees and the universities that receive the most federal money. Overall about 75% of NSF funding goes to academic institutions.

Even when controlling for other factors, we find that for each additional employee a university has serving on an NSF advisory committee that university can expect to see an additional $125,000 to $138,000 in NSF funding.

Although the 144 NSF advisory committees do not make funding decisions directly, they do “review and provide advice on program management, overall program balance, and other aspects of program performance,” according to the NSF.

At a big picture view, looking at the data on NSF grant awards and NSF advisory committee representation reinforces just how much of the money and representation is concentrated in a limited number of major universities.

Twenty percent of top research universities got 61.6% of the NSF funding going to top research universities between 2008 and 2011. These universities also had 47.9% of the representatives on NSF advisory committees who came from top research universities during the same period. The next 20% of universities got 21.9% of the funding, and had 25.7% of the representatives. The bottom 20% research universities had just 1.0% of the funding and have 2.4% of the representatives.

Just 23 universities account for more than half of the funding awarded by the NSF top to research universities. See Table 1 .

The University of California tops the list by far, because we combined all University of California campuses (due to data issues, see our data and methodology section), followed by Cal Tech, the University of Illinois, Michigan and Cornell. Interestingly, of the traditional top three universities (Harvard, Princeton and Yale), only Harvard shows up on the above list, at No. 22.

For complete data on 171 major research universities, click here. (The 171 universities come from the US News and World Report list of 200 major research universities. We selected only universities that had some interaction with the NSF between 2008 and 2011).

More representatives on advisory committees, more funding

Figure 1 plots the average NSF funding level for the university from 2008-2011, and the average number of representatives serving on NSF committees during this same period.

The correlation is clear. The more university-affiliated individuals serve on NSF advisory committees, the more NSF funding the university gets. Mostly, big state schools, with a few Ivy League schools in the mix, dominate the higher echelons of funding and representation. Interestingly, both Cal Tech and M.I.T., two of the pre-eminent research institutions in the country, get substantial NSF funding with limited representation. (Note: The University of California is left off this chart since it is a far outlier on both average funding ($361 million) and average representation (638.5 members). Because the quality of our data prevents us from breaking down the University of California by campus, we largely omit it from our analysis.)

A second scatterplot (Figure 2) examines the relationship between the number of committees and the funding levels. Here the data take on a slightly different relationship. With the exception of a few outliers, there is a changing relationship between the diversity of committees and the NSF funding levels.  It is more exponential than linear. Having representation on just a few committees doesn’t consistently correlate with higher funding, but having representation on a lot of committees is strongly correlated with higher funding.

Do more representatives help universities secure more funding?

The NSF “strives to conduct a fair, competitive, transparent, merit-review process for the selection of projects,” based on intellectual merit and broader impacts. Each year, the NSF produces an annual report on the merit review process. To make funding decisions, the NSF relies on tens of thousands of expert reviewers, though program officers make the final decisions.

Advisory committees oversee the general direction of the NSF program areas, including identifying “disciplinary needs and areas of opportunities.” As for who gets on these committees, the NSF explains that: “Many factors are weighed when formulating Committee membership, including the primary factors of expertise and qualifications, as well as other factors including diversity of institutions, regions, and groups underrepresented in science, technology, engineering, and mathematics.”

An example of such a committee is the Proposal Review Panel for Information and Intelligent Systems. Following the hyperlink provided would take you to a list of committee members in Influence Explorer, most of whom have university affiliations.

Showing that more representatives help universities get more funding than they would otherwise have received is difficult. There is a very good and reasonable explanation for the patterns we observe in the two above scatter plots: The NSF tries to get the most knowledgeable experts and accomplished academics to serve on its committees. Not surprisingly, the universities that attract the most NSF money are also likely to be home to many accomplished experts, since they are all leading research universities.

However, there are a few ways in which representatives could help their own universities to improve their chances. One possibility is that if a department has a representative on an NSF committee, that representative will be able to pass along funding opportunities and advice on navigating on the decision-making process of the committee to others in the university, thus strengthening others’ chances. Insiders can help others to better understand what a review committee might be looking for.

Another possibility is that in directing the general funding strategies of NSF program areas, advisory committees might see what their universities are doing as particularly valuable. Or more benignly, they might be more aware of the cutting-edge research within their universities just because it is being done by colleagues they interact with on a regular basis.

One way to investigate the relationship is to do a regression analysis, which allows us to control for different factors simultaneously. For those of a more technical mind, the details are below. For those who want the quick takeaway, it goes like this: Controlling for previous NSF funding and university endowment, universities with more NSF advisory committee representatives get more NSF funding than those that don’t. Each additional representative translates into about an extra $125,000 to $138,000 in NSF funding, controlling for other factors. The number of representatives is more important than the number of committees with representatives. Lobbying expenditures make no difference.

The entire article with the tables is very worth reading.

Are Some Climate Scientists And Program Managers At NSF Gaming the System With Respect To Providing Inadequately Assessed Multi-Decadal Predictions Of Changes In Climate Statistics To The Impacts Community?

Are climate scientists and program managers who fund them, who  misrepresent multi-decadal climate model predictions to the impacts communities as skillful projections,  “gaming the system” ?  Gaming the system is defined as

Gaming the system (or bending the rules, playing the system, abusing the system, milking the system or working the system) can be defined as “[using] the rules and procedures meant to protect a system in order, instead, to manipulate the system for [a] desired outcome”.

My answer is YES, unless these individuals can demonstrate skill at multi-decadal regional climate predictions which they fund. To my knowledge, however, this demonstration of skill has not been shown. Thus those individuals are either very naive or are deliberately gaming the system.

I am quite blunt in this post (and see also this post ) where I focus on the NSF, since the individuals I am referring to have been alerted to the failings with respect to providing multi-decadal predictions of changes in regional and local climate statistics to the impacts community.  To be clear, my criticism is not with all of NSF, but only with those program managers who are funding multi-decadal impact studies based on predictions of changes in regional and local climate statistics.

Despite the identification of this issue and either refuting our findings, or accepting them and not continuing to fund, they have ignored this issue.  I see no other explanation for them ignoring our findings, except that they can use their claims of providing forecasts for the policymakers and stakeholders in order to continue the flow of dollars from the federal treasury into their programs.

At best this is short-sighted, and at worse it is dishonest.

As I discussed in my post

Short Circuiting The Scientific Process – A Serious Problem In The Climate Science Community

There has been a development over the last 10-15 years or so in the scientific peer-reviewed literature that is short circuiting the scientific method.

The scientific method involves developing a hypothesis and then seeking to refute it. If all attempts to discredit the hypothesis fails, we start to accept the proposed theory as being an accurate description of how the real world works.

A useful summary of the scientific method is given on the website sciencebuddies.org.where they list six steps

• Ask a Question
• Do Background Research
• Construct a Hypothesis
• Test Your Hypothesis by Doing an Experiment
• Analyze Your Data and Draw a Conclusion
• Communicate Your Results

Unfortunately, in recent years papers have been published in the peer-reviewed literature that fail to follow these proper steps of scientific investigation. These papers are short circuiting the scientific method.

Many of these peer-reviewed papers are funded by the NSF.

As of today’s date, it is clear they are still ignoring addressing the issues that we have summarized in our peer-reviewed articles – Pielke and Wilby 2012 and Pielke et al 2012. They are gaming the system in order to continue the high level of funding for impact studies that are based on the multi-decadal regional and local climate model predictions of changes in climate statistics.

While I support the NSF funding of the assessment of predictability; e.g. see below for an example from KNMI in the Netherlands

Guest Post Titled “Decadal Prediction Skill In A Multi-Model Ensemble” By Geert Jan van Oldenborgh, Francisco J. Doblas-Reyes, Bert Wouters, Wilco Hazeleger

Seminar Announcement – On The Reliability Of Climate Models: How Well Do They Describe Observed Trends? By Geert Jan van Oldenborgh Of KMNI

The Difference Between Prediction and Predictability – Recommendations For Research Funding Related to These Distinctly Different Concepts

the provision of multi-decadal regional and local climate model results for the coming decades and representing them as skillful projections is scientifically invalid. Program managers, principal investigators, policymakers and others who report the results as skillful and do not include the needed disclaimer that I presented in my post

Climate Science Malpractice – The Promotion Of Multi-Decadal Regional Climate Model Projections As Skillful

are certainly gaming the system (or are very naive).

Remarkably, as I have learned, there is no accountability and review of what the NSF program managers fund in terms of topic area, including no effective mechanism to contest if they are actually funding robust scientifically tested research.

They do not even need to retain copies of their e-mails to ascertain how they handle issues such as raised in our papers and in my weblog posts, but can delete them so they cannot be obtained under an FOIA request. I discussed this failing in my post

My Experiences With A Lack Of Proper Diligence And Bias In The NSF Review Process For Climate Proposals

where I concluded that

• NSF does not retain a record of e-mail communications
• NSF is cavalier in terms of the length of time proposals are under review.
• NSF has decided to emphasize climate modeling and of funding multi-decadal climate predictions, at the expense of research which can be tested against real-world observations.
• NSF penalizes scientists who criticize their performance.

With respect to e-mails (and this part of NSF accountability),  as was communicated to me by a lawyer at the NSF

On Fri, 15 Apr 2011, Jensen, Leslie A. wrote:

Dear Sir:

A proper search has been accomplished for all named individuals.  Email is  not a permanent record and meetings from several years ago would be deleted.  The Foundation’s email retention policy is repeated below:

———————————-

Exchange Server: Most users have their mail delivered to their Exchange  Server mailbox. If you haven’t done anything special, that is where your  mail is delivered and stored. In Outlook your mailbox is the folder that  includes your name in the folder name (top folder in your Folder List). The  Exchange Servers are backed up to tape nightly, and the tapes are retained  for 14 days, then destroyed. Exchange Server has a feature that allows you  to recover deleted messages (even after the trash is emptied). That feature  is set to retain deleted messages for 5 days. When these features are combined, it means that 19 days after you delete a message and empty the  trash, nobody can recover it.

This lack of accountability with federally supported NSF should be of concern to everyone, regardless of your perspective on the climate science issue.

In my post

Climate Assessment Oligarchy – The IPCC

I wrote with respect to the IPCC

An oligarchy is a

“form of government in which all power is vested in a few persons or in a dominant class or clique; government by the few.”

This definition can certainly also be applied to  a number of program managers at the NSF, and other funders, of climate science research. Until and unless a new direction that is actually based on the scientific method is introduced, we will continue to see this abuse of their positions (and waste of funds) as stewards of funding scientific research.

source of image

Follow Up On My Post “Funding Agency Bias – A Short Summary”

Don Bishop, in response to the post

Funding Agency Bias – A Short Summary

sent the informative e-mail below which I invited him to share with everyone. It is reproduced with his approval.

Date: Tue, 27 Mar 2012 11:45:13 -0400 (EDT) From: Don Bishop Subject: Agency funding bias

Hi Roger

While reading today’s post [that of March 27 2012] about bias, and how models do not give useful regional weather/climate predictions, I was reminded of Judy Curry’s presentation in Boulder on August 31, 2011 – an example of what is needed.

The workshop was sponsored by NOAA on water cycle science challenges, and Curry’s presentation showed how lower 48 droughts relate to PDO and AMO, what  was predicted for those ocean cycles, and therefore what the drought predictions were: the analogue for the next decade or so was the 1950s.

_http://judithcurry.com/2011/08/31/water-too-little-too-much-part-ii/_ (http://judithcurry.com/2011/08/31/water-too-little-too-much-part-ii/)

Her slides:

_http://www.esrl.noaa.gov/psd/events/2011/pdf/water-cycle-presentations/Curr y_noaaWaterClimate.pdf_ (http://www.esrl.noaa.gov/psd/events/2011/pdf/water-cycle-presentations/Curry_noaaWaterClimate.pdf)

Don Bishop

The NSF CREATIV Initiative On Interdisciplinary Research – Another Example Of Thinking Inside The Box

Several months ago, the NSF announced the CREATIV initiative in  a Dear Colleague letter.  Since we have proposed a broader, interdisciplinary  approach to the assessment of risks to key societal and environmental risks in our paper

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2011: Dealing  with complexity and extreme events using a bottom-up, resource-based  vulnerability perspective. AGU Monograph on Complexity and  Extreme Events in Geosciences, in press

I submitted an application to submit based on two issues:

1. The assessment of the skill at the top-down global climate model predictions of changes in climate statistics that are relevant to stakeholders in terms of the metrics they need

and

2. The quantification of the climate and other threats to these metrics with respect to selected key resources in water, food, energy, human health and ecosystem function.

My experience, that I have documented below, with the  program managers who handled this issue illustrates the problem with obtaining funding at the NSF in climate related studies unless you fit into their particular area of interest (i.e. there are “favored topics” despite what is announced in the Dear Colleague letter).

The CREATIV program itself is described as

CREATIV (Creative Research Awards for Transformative Interdisciplinary Ventures): a pilot grant mechanism under the Integrated NSF Support Promoting Interdisciplinary Research and Education (INSPIRE) initiative, to support bold interdisciplinary projects in all NSF-supported areas of science, engineering, and education research.

The CREATIV program has these claimed goals

• Create new interdisciplinary opportunities that are not perceived to exist presently.
• Attract unusually creative high-risk / high-reward interdisciplinary proposals.
• Provide substantial funding, not limited to the exploratory stage of the pursuit of novel ideas.
• Designate no favored topics; be open to all NSF-supported areas of science, engineering, and education research.

In terms of “no favored topics” they write

No. There are no  favored topics. In terms of review  criteria, unusual promise for societal benefit can contribute to the broader  impacts of a proposal.

In the following, I have reproduced the e-mails involved in my interaction with the program managers at the NSF.  I have redacted their names, since these individuals just illustrate a culture at the NSF in the area of climate science that is clearly biased to perpetuating a particular viewpoint on the climate issue.

My First E-mail

Dear sir/madam

Well over a month ago, I submitted the short write up at the NSF website (using your form – http://www.nsf.gov/od/oia/creativ/inquiry.cfm) on submitting a CREATIV proposal, but have not heard anything regarding my submission.

Please let me know who I should contact regarding its status. CREATIV is described at

http://www.nsf.gov/pubs/2012/nsf12011/nsf12011.jsp

and was announced by Professor (NSF Director) Suresh, who I have copied to on his MIT e-mail address.

I submitted my information as requested in the text

“Potential proposers are encouraged to begin the process by submitting the CREATIV Inquiry Data Form, as explained on the FAQ page. Before writing and submitting a CREATIV proposal, it is the principal investigator’s responsibility to obtain written authorization to submit a CREATIV proposal by NSF program directors from at least two intellectually distinct divisions or programs.”

I completed this step [from http://www.nsf.gov/pubs/2012/nsf12012/nsf12012.jsp#tdcl]

“As a first step, you are encouraged to fill out and submit the CREATIV Inquiry Data Form, here. Before submitting the form, you must identify at least one appropriate NSF Program Director to consider your inquiry.”

We listed a number of NSF Program Directors to assess and based our request on our interdisciplinary paper

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2012: Dealing with complexity and extreme events using a bottom-up, resource-based vulnerability perspective. AGU Monograph on Complexity and Extreme Events in Geosciences, in press.

https://pielkeclimatesci.files.wordpress.com/2011/05/r-365.pdf

However, so far, we have had no feedback from the NSF on our Inquiry submission.

Sincerely

Roger A. Pielke Sr.

My Response: Here is the first response which I received only after I copied the Director of the NSF.

Dear Dr. Pielke,

I am an NSF program officer.  I was not among the program officers that you listed on your CREATIV inquiry, but I am among those to whom your inquiry was routed.

It seems that perhaps the following NSF grants are already funding aspects of what you are proposing:

A. RESIN Grants from ENG/EFRI (RESIN = Resilient and Sustainable Infrastructure).  One example:

EFRI-RESIN: Assessing and Managing Cascading Failure Vulnerabilities of Complex, Interdependent, Interactive, Adaptive Human-based Infrastructure Systems http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0836047

B. WSC Grants (WSC = Water Sustainability and Climate).  Some examples:

WSC-Category 2: Extreme events impacts on water quality in the Great Lakes: Prediction and management of nutrient loading in a changing climate http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1039043

Collaborative Research, WSC-Category 2: Regional Climate Variability and Patterns of Urban Development – Impacts on the Urban Water Cycle and Nutrient Export

http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1058038

C. RCN-SEES Grants (RCN-SEES = Research Coordination Networks – Science, Engineering, and Education for Sustainability).  Some examples:

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1140196

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1140384

So, I do not feel that the CREATIV path is a fit for your concept.  One path forward might be for you to put together an RCN-SEES proposal that would be built around your approach but would draw upon related ongoing NSF-funded research and researchers such as are described at the links above.  The next RCN-SEES deadline is already posted (Feb. 4, 2013).  Here is a link:

http://www.nsf.gov/pubs/2011/nsf11531/nsf11531.htm

Another option might be to pursue CaMRA, as described in NSF’s posted FY 2013 Budget Request: “Creating a More Disaster-Resilient America (CaMRA) aims to catalyze basic research and education efforts in hazard-related science, engineering, risk assessment and decision making in order to improve forecasting and prediction of natural and technological hazards, mitigate their effects, and prepare communities to respond to, and recover from disasters.”

Best wishes,

XXXX

My Reply

Hi XXXX

Thank you for your detailed and thoughtful reply.

These programs certainly would fit with out bottom-up focus, but a deadline of February 2013 for the next RCN-SEES deadline (Feb. 4, 2013) for just submitting, means it would be 1 1/2 years or more from now before we would have any funding, even if we were successful in a proposal.

Since our research; as presented in our papers

Pielke Sr., R.A., R. Wilby, D. Niyogi, F. Hossain, K. Dairuku, J. Adegoke, G. Kallos, T. Seastedt, and K. Suding, 2012: Dealing with complexity and extreme events using a bottom-up, resource-based vulnerability perspective. AGU Monograph on Complexity and Extreme Events in Geosciences, in press. https://pielkeclimatesci.files.wordpress.com/2011/05/r-365.pdf

Pielke Sr., R.A., and R.L. Wilby, 2012: Regional climate downscaling – what’s the point? Eos Forum, 93, No. 5, 52-53, doi:10.1029/2012EO050008. https://pielkeclimatesci.files.wordpress.com/2012/02/r-361.pdf

shows that the current approach of regional and local downscaling of multi-decadal global model climate predictions for use to determine vulnerabilities to key societal and environmental resources is seriously flawed, we feel it is important to examine the value of the bottom up approach. The IPCC type scenarios are, at best, a subset of what climate threats that will be faced in the future.

The CREATIV option seems to be the only option to use in the near term. Please advise, however, if the programs such as

EFRI-RESIN: Assessing and Managing Cascading Failure Vulnerabilities of Complex, Interdependent, Interactive, Adaptive Human-based Infrastructure Systems

WSC Grants (WSC = Water Sustainability and Climate).

Collaborative Research, WSC-Category 2: Regional Climate Variability and Patterns of Urban Development – Impacts on the Urban Water Cycle and Nutrient Export

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

have any closer deadlines to submit [I assume the last two have a Feb 2013 date for new submissions).

Sincerely

Roger

The Response:

 Hello Roger,

My colleague YYYY has kindly provided information on WSC below.

There is some chance that EFRI-RESIN may have another solicitation, but that is not yet decided, and, at any rate, the deadline would be after the next RCN-SEES deadline of February 4, 2013.

In my message below (with relevant links), the following two are already existing NSF RCN-SEES grants that I was suggesting may have a relationship to your concept (they are not funding opportunities separate from RCN-SEES with the deadline of February 4, 2013):

RCN-SEES: Climate, Energy, Environment, and Engagement in Semi-Arid Regions (CE3SAR)

RCN – SEES: Sustainable Cities – People and the Energy-Climate-Water Nexus

The Response from the other NSF Program Officer:

Roger:

The Water Sustainability and Climate (WSC) competition is likely to have its next deadline in around 18 months, although this is uncertain and the specific requirements for proposals may change. The current solicitation for large awards is funding research that focuses intensely on one watershed or that compares water sustainability issues in two or more watersheds. The solicitation requires that proposals integrate the work of experts from fields in engineering, geosciences, biosciences, and social sciences. Some of what you want to do may well fit with WSC.

Best wishes, YYYY

My Response

Hi YYYY

Thank you for your feedback. What I am interested in is not large funding, but about 1750K to 200K per year for three years to further examine:

i) shortcomings of the top-down (IPCC; CCSP) (outcome) vulnerability approach and,

ii) provide examples of the more holistic, inclusive assessment of vulnerabilities in our bottom-up (contextual) approach.

Since I am leading, as Editor-in-Chief, a 5 volume set of books which introduce this perspective (and provides examples), I would be able to leverage from the work that we are producing in this volume and to work with a number of those colleagues to assist in building on it. One issue with the set of 5 books is that there remains considerable existing emphasis on the narrower top-down, global climate model dominated view, and a focused study on the bottom-up approach is needed.

My Editors are Faisal Hossain on water resources (Tennessee Technological University); Jimmy Adegoke and Caradee Wright on human health (CIRS; South Africa); Tim Seasteadt and Katie Suding on ecosystem function (University of Colorado/ UC Berkeley); Dev Niyogi on food (Purdue); and George Kallos on energy (University of Athens, Greece).

My proposal is very straightforward. It is to determine the major threats to a selected examples of local and regional water, food, energy, human health, and ecosystem function resources to short term weather events (e.g. tornadoes), multi-weekly and seasonal long events (droughts) and multi-year and decadal variability and change (e.g. an increase or decrease in the 30 year warm season maximum temperatures), and also from other social and environmental issues. After these threats are identified for each resource, then the relative risks from weather and climate issues can be compared with other risks in order to adopt optimal preferred mitigation/adaptation strategies.

The questions to be asked to a selected group of stakeholders (chosen by working with those who worked with me as Editors) are:

1. Why is this resource important? How is it used? To what stakeholders is    it valuable?

2. What are the key environmental and social variables that influence this    resource?

3. What is the sensitivity of this resource to changes in each of these    key variables? (This includes, but is not limited to, the sensitivity    of the resource to climate variations and change on short (days);    medium (seasons) and long (multi-decadal) time scales).

4. What changes (thresholds) in these key variables would have to    occur to result in a negative (or positive) outcome for this resource?

5. What are the best estimates of the probabilities for these changes to    occur? What tools are available to quantify the effect of these    changes? Can these estimates be skillfully predicted?

6. What actions (adaptation/mitigation) can be undertaken in order to    minimize or eliminate the negative consequences of these changes (or to    optimize a positive response)?

7. What are specific recommendations for policymakers and other    stakeholders?

I hope you can direct me to where I can seek funding for this effort prior to 2013.

Sincerely

Roger

The Response 

Roger:

To the extent that your project is descriptive and essentially a literature review, the project is probably not appropriate for NSF whose mission is to support basic research.

If elements of your project are likely to produce a theoretical or methodological advance, then the project is probably appropriate for the DRMS Program. The next target date is August 18 with funding decisions in November and December.

Best wishes,

YYYY

My Reply

Hi YYYY

The approach is quite a bit more than a literature survey, as stakeholders would need to be interviewed and data analyzed. It is a more robust approach the current top-down method, as we concluded in our AGU book chapter and the EOS article. The top-down approach is currently receiving quite large amounts of NSF funding, yet the basic science robustness of this approach has not been adequately vetted.  This would be part of our research.

I do see under list of activities for the Decision, Risk and Management Sciences (DRMS) -https://pielkeclimatesci.wordpress.com/wp-admin/edit.php

that you fund small grants that are high-risk and of a potentially transformative nature (EArly-Concept Grants for Exploratory Research – EAGER).

Our approach would be to test if the current top-down global climate model multi-decadal prediction approach is fatally flawed as a major driver of impact studies(as we have found so far in our studies), and that funds are being wasted on this methodology. This would be a transformative finding if our finding is confirmed.

To my knowledge, there is no coherent NSF program that is looking into this issue.

Best Regards

Roger

Their Reply

Roger:

The DRMS proposal officers have concluded that your proposed project does not seem to be sufficiently risky and transformational to warrant DRMS encouraging you to submit an EAGER proposal. The notion that one can usefully look at the incremental threat to a sector from any particular hazard is not a great conceptual leap forward. There is no reason your proposed project should not undergo NSF’s peer review procedures.

Best wishes,

YYYY

My Reply

YYYY

Where in the NSF (or elsewhere for that matter) is the assessment of the skill of the IPCC-type models in the prediction of changes in multi-decadal regional climate statistics being completed? It is these model results that are being given to the impacts communities. If our conclusions, as summarized in the peer reviewed articles I sent to you, are correct, NSF is wasting a lot of money. It would seem this is more than an incremental issue. There are two parts to what I have proposed, and this assessment of climate prediction skill is the first part. I am familiar, and support the assessment of the decadal predictability initiative of the NSF (as an initial condition), but this is distinct from the use of the multi-decadal climate change predictions. Best Regards Roger

My Second Reply since I was not getting a response

YYYY

Do you plan to reply to my request for information as to which NSF funded projects are assessing the skill of multi-decadal regional climate predictions?

The regional climate model predictions for the coming decades that are being provided to the impacts communities is not basic science unless their predictive skill can be determined.

To my knowledge, however [and as we summarized in the Pielke and Wilby, 2012 EOS article], they have not been shown to have skill even in a hindcast mode when compared with observed variations and long term changes in climate statistics.

This assessment is very much needed (unless you or others provide information that refutes my finding) and is one of the two pillars of my proposed study. This, as I have written, is hardly an incremental study, but underpins a large amount of spending by the NSF.

I also plan to post regarding this issue on my weblog in the coming days, but would like your feedback first on my question.

Best Regards

The Response:

Roger:

The appropriate responder to your question is in the Geological Sciences Directorate, not the Social, Behavioral and Economic Sciences Directorate. I do not know if NSF is funding projects that are assessing the skill of multi-decadal regional climate predictions.

Best wishes,

YYYY

My Reply

YYYY

Thank you for the quick reply.

Please let me know who I should contact in the Geological Sciences  Directorate.

In terms of the Social, Behavioral and Economic Directorate, however, I am under the understanding that climate impact studies on society are funded  which use (and have their fundamental basis on) the regional climate model  projections. If my understanding is incorrect on this, please let me know.

Best Regards

Roger

Their Reply

Roger:

One good way to find out what NSF is funding is to check out abstracts of  awards listed on the NSF website (www.nsf.gov). You can search by keyword,  program, etc. The website also describes the different programs in the  atmospheric sciences division of the geosciences directorate.

Best wishes,

YYYY

My Reply

YYYY

We have already done that and there are no grants that we found that assess the model skill as I have outlined in our papers. They are quite well aware of the issues I have raised on this subject in the Geosciences Directorate and have chosen to ignore them. The only exception is their new focus on decadal predictability, which, however, is still distinct from the multi-decadal impact studies based on the IPCC type projections.

My point of seeking funding outside is

i) that programs that examine societal and environmental vulnerability are using the results from the multi-decadal global climate model predictions as an essential part of their studies. These results are then being used (misused in my view) by the policy and political communities.

and

ii) The Geosciences Directorate has chosen to arbitrarily accept the model predictions as applied to impact studies as robust. They have ignored, so far, attempts to get them to properly assess this issue.

My bottom-line conclusion is that there is really no basic science aspect to these NSF funded impact studies [which appear across the NSF funding spectrum], yet NSF continues to fund them.

Best Regards

Roger

Their Reply       None

source of image

NSB/NSF Seeks Input on Proposed Merit Review Criteria Revision and Principles

The National Science Board has sent out a notice requesting input on the NSF review process. Their request is reproduced in this post. One glaring issue that is missing is accountability. I discussed this subject in my posts

My Experiences With A Lack Of Proper Diligence And Bias In The NSF Review Process For Climate Proposals

Is The NSF Funding Untestable Climate Predictions – My Comments On A $6 Million Grant To Fund A Center For Robust Decision–Making On Climate And Energy Policy”

The National Science Foundation Funds Multi-Decadal Climate Predictions Without An Ability To Verify Their Skill

NSF Decision On Our Request For Reconsideration Of A Rejected NSF Proposal On The Role Of Land Use Change In The Climate System

Is The NSF Funding Process Working Correctly?

I have made the following recommendations:

• Guarantee that the review process be completed within 6 months [my most recent land use and climate proposal was not even sent out for review until 10 months after its receipt!)
• Retain all e-mail communications indefinitely (NSF staff can routinely delete e-mails, such that there is no record to check their accountability)
• Require external independent assessments, by a subset of scientists who are outside of the NSF, of the reviews and manager decisions, including names of referees. This review should be on all accepted and rejected proposals ( as documented in the NSF letter at the end of this post, since they were so late sending out for review, they simply relied on referees of an earlier (rejected) proposal; this is laziness at best).

The National Science Board request follows. I will be submitting my comments, based on the above text, and urge colleagues who read my weblog to do likewise.

NSB-11-42
NSB/NSF Seeks Input on Proposed Merit Review Criteria Revision and Principles

National Science Board
June 14, 2011

Over the past year, the National Science Board (NSB) has been conducting a review of the National Science Foundation’s merit review criteria (Intellectual Merit and Broader Impacts). At the Board’s May 2011 meeting, the NSB Task Force on Merit Review proposed a revision of the two merit review criteria, clarifying their intent and how they are to be used in the review process. In addition, the Task Force identified a set of important underlying principles upon which the merit review criteria should be based. We now seek your input on the proposed revision and principles.

The Task Force looked at several sources of data for information about how the criteria are being interpreted and used by the NSF community, including an analysis of over 190 reports from Committees of Visitors. The Task Force also reached out to a wide range of stakeholders, both inside and outside of NSF, to understand their perspectives on the current criteria. Members of NSF’s senior leadership and representatives of a small set of diverse institutions were interviewed; surveys about the criteria were administered to NSF’s program officers, division directors, and advisory committee members and to a sample of 8,000 of NSF’s Principal Investigators (PIs) and reviewers; and the NSF community at large was invited to provide comments and suggestions for improvements through the NSF web site ( http://www.nsf.gov/nsb/publications/2011/01_19_mrtf.jsp). The stakeholder responses were very robust—all told, the Task Force considered input from over 5,100 individuals.

One of the most striking observations that emerged from the data analyses was the consistency of the results, regardless of the perspective. All of the stakeholder groups identified similar issues, and often offered similar suggestions for improvements. It became clear that the two review criteria of Intellectual Merit and Broader Impacts are in fact the right criteria for evaluating NSF proposals, but that revisions are needed to clarify the intent of the criteria, and to highlight the connection to NSF’s core principles.

The two draft revised criteria, and the principles upon which they are based, are below. Comments are being collected through July 14—we invite you to send comments to meritreview@nsf.gov. It is expected that NSF will develop specific guidance for PIs, reviewers, and NSF staff on the use of these criteria after the drafts are finalized. Your comments will help inform development of that guidance, and other supporting documents such as FAQs.

The Foundation is the primary Federal agency supporting research at the frontiers of knowledge, across all fields of science and engineering (S&E) and at all levels of S&E education. Its mission, vision and goals are designed to maintain and strengthen the vitality of the U.S. science and engineering enterprise and to ensure that Americans benefit fully from the products of the science, engineering and education activities that NSF supports. The merit review process is at the heart of NSF’s mission, and the merit review criteria form the critical base for that process.

We do hope that you will share your thoughts with us. Thank you for your participation.

Ray M. Bowen Chairman, National Science Board

Subra Suresh Director, National Science Foundation

Merit Review Principles and Criteria
The identification and description of the merit review criteria are firmly grounded in the following principles:

1. All NSF projects should be of the highest intellectual merit with the potential to advance the frontiers of knowledge.
2. Collectively, NSF projects should help to advance a broad set of important national goals, including:
   • Increased economic competitiveness of the United States.
   • Development of a globally competitive STEM workforce.
   • Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
   • Increased partnerships between academia and industry.
   • Improved pre-K–12 STEM education and teacher development.
   • Improved undergraduate STEM education.
   • Increased public scientific literacy and public engagement with science and technology.
   • Increased national security.
   • Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships.
3. Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research. All are valuable approaches for advancing important national goals.
4. Ongoing application of these criteria should be subject to appropriate assessment developed using reasonable metrics over a period of time.

Intellectual merit of the proposed activity

The goal of this review criterion is to assess the degree to which the proposed activities will advance the frontiers of knowledge. Elements to consider in the review are:

1. What role does the proposed activity play in advancing knowledge and understanding within its own field or across different fields?
2. To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
3. How well conceived and organized is the proposed activity?
4. How well qualified is the individual or team to conduct the proposed research?
5. Is there sufficient access to resources?

Broader impacts of the proposed activity

The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national goal(s). Elements to consider in the review are:

1. Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling description of how the project or the PI will advance that goal(s)?
2. Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
3. Is the rationale for choosing the approach well-justified? Have any innovations been incorporated?
4. How well qualified is the individual, team, or institution to carry out the proposed broader impacts activities?
5. Are there adequate resources available to the PI or institution to carry out the proposed activities?

Predicting Wars Based On Climate Change – Really?

There is a remarkable recently NSF funded study that claims to be able predict the liklihood of wars based in part on multi-decadal climate change. Readers can assess from their own perspectives if this is appropriate as a scientifically testable study. 

In my view, a significant portion of NSF funding is going to projects that involve predictions decades into the future, and this is just one example.

 These predictions, of course, cannot be verified during the lifetime of the project (and even for years afterwards), so it is not clear to me why such research is being funded by the NSF. 

Here is the study with an extract from the abstract.  

Collaborative Research: Climate Change and Variability and Armed Conflicts in Africa South of the Sahara

“The number of armed conflicts has declined after the end of the Cold War. There is also a long-term trend towards less severe armed conflicts, though climate change threatens to reverse this favorable trend……Using a predictive model of the coupled natural (climate) and social (violence) systems, with feedback loops and mediating socio-political-economic variables, the PIs will measure the impact of adverse climate change and/or changes in climate variability on the rate of armed conflict, determine which mediating factors influence the rate of this impact, and project the violence outcomes on the basis of different climate change/variability scenarios.”

NSF Decision On Our Request For Reconsideration Of A Rejected NSF Proposal On The Role Of Land Use Change In The Climate System

On May 18 2010 I posted on a proposal to NSF that was highly rated each time it was submitted, but was rejected each of the three times it was submitted after further revisions were made. 

The May post is

Is The NSF Funding Process Working Correctly?

The NSF mission reads

The National Science Foundation Act of 1950 (Public Law 81-507) set forth NSF’s mission and purpose:

To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense….

The Act authorized and directed NSF to initiate and support:

• basic scientific research and research fundamental to the engineering process,
• programs to strengthen scientific and engineering research potential,
•  science and engineering education programs at all levels and in all the various fields of science and engineering,
• programs that provide a source of information for policy formulation,
• and other activities to promote these ends.

Over the years, NSF’s statutory authority has been modified in a number of significant ways. In 1968, authority to support applied research was added to the Organic Act. In 1980, The Science and Engineering Equal Opportunities Act gave NSF standing authority to support activities to improve the participation of women and minorities in science and engineering. Another major change occurred in 1986, when engineering was accorded equal status with science in the Organic Act.

NSF has always dedicated itself to providing the leadership and vision needed to keep the words and ideas embedded in its mission statement fresh and up-to-date. Even in today’s rapidly changing environment, NSF’s core purpose resonates clearly in everything it does: promoting achievement and progress in science and engineering and enhancing the potential for research and education to contribute to the Nation. While NSF’s vision of the future and the mechanisms it uses to carry out its charges have evolved significantly over the last four decades, its ultimate mission remains the same.

The title and Project Summary of our rejected proposal is

“Collaborative Research: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes since European Settlement”

with the Project Summary

“The Earth’s weather and climate is strongly influenced by the properties of the underlying surface. Much of the solar energy that drives the atmosphere first interacts with the land or sea surface. Over land regions this interaction is modulated by surface characteristics such as albedo, aerodynamic roughness length, leaf area index (LAI), etc. As these characteristics change, either from anthropogenic or natural land-cover disturbances, the amount of energy reaching the atmosphere from the land surface, and thus weather and climate, is expected to change. The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System (RAMS) coupled with the Simple Biosphere (SiB) model, SiB-RAMS, will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox (2008). The influence of the land-cover changes on temperature and precipitation will be examined and compared with that expected from CO2-induced climate change (IPCC 2007). The seasonality of the changes in precipitation and temperature due to land-cover change will be explored. Also, the relative importance of each land-cover biophysical parameter to the total simulated change in temperature and precipitation will be assessed.”

We have presented the letter from the Deputy Assistant Director regarding our request for reconsideration.

This letter is quite informative as it is a cursory, pro forma response without any detail. What it confirms is that program managers have considerable latitude in decision-making and can eliminate well reviewed projects if they differ from their priorities. The program managers decide what is “basic scientific research and research fundamental to the engineering process” rather than relying on the reviewers to determine this [of course, they can also select known biased reviewers if they want to reject a proposal].

Since the level of ratings of our proposal were high, the reason for the rejection is based on the program managers concluding that the role of land use change in the climate system is not a high research priority.  Also, despite the NSF requirement listed their mission statement “to support activities to improve the participation of women and minorities in science and engineering”,  the fact that woman (Dr. Lixin Lu) was the PI on the project was not discussed in the reconsideration.

My recommendation to improve the process, which I presented in my May post is

present ALL proposal abstracts, anonymous reviews of both accepted and rejected proposals and program managers decision letters (or e-mails) on-line for public access

present the date of submission and final acceptance (or rejection) of the proposal.

I also recommend they make easily available the list of all of the reviewers used during the year within each NSF program office.

NSF program managers have considerable ability to slant research that they fund with insufficent transparency of the review process. This has become quite a problem in the climate science area where, as one example, in recent years they have elected to fund climate predictions decades into the future (e.g. see which was funded in part by the NSF; I will discuss specific examples of such funded projects by the NSF in a future post).

Is The NSF Funding Process Working Correctly?

This post is titled “Is The NSF Funding Process Working Correctly?”

ABSTRACT

The short answer to this question is NO. I recommend the following changes in the NSF review procedures to improve the process:

present ALL proposal abstracts, anonymous reviews of both accepted and rejected proposals and program managers decision letters (or e-mails) on line for public access

present the date of submission and final acceptance (or rejection) of the proposal.

 

TEXT OF MY POST

From my experiences in recent years, in terms of climate science issues, the answer is NO. Indeed, while I will discuss just one (rejected) proposal below, I also have a proposal still under consideration that was submitted on May 13 2009 (an NSF-accomplishment based proposal) but only sent out for review in late March 2010! I will update on my weblog the disposition of that proposal when it is finally decided on).

My experience with the NSF funding process of climate science in recent years fits with Dick Lindzen’s summary (see)

“In brief, we have the new paradigm where…. government largely determines the nature of scientific activity…..”

Perhaps my experience is unique, although from discussions with colleagues who have been visible in questioning the IPCC perspective of climate change, there are others who have had similar treatment in the NSF funding process.

In the following text, I document  my experiences with a climate science proposal (since we kept the e-mail documentation, the information is readily available). 

I will summarize the situation first, and provide more detail later in the post as an Appendix.

NSF Proposal: “Collaborative Research: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes since European Settlement” [with Lixin Lu of Colorado State University and myself at the University of Colorado at Boulder]

This proposal, modified each time in response to the reviews with a change of PIs after the first submission [since funding was lost for the PI of the first submission and he had to leave my research group as a result (John Strack)] has been rejected three times.  The evaluations each time were

1. Submission 1  3 VERY GOODs

2. Submission 2 2 VERY GOODs; 1 GOOD; 1 FAIR

3. Submission 3 3 VERY GOODs

The NSF Review Rating Scale (slide 17) is

• Excellent: Outstanding proposal in all respects; deserves highest priority for support
• Very Good: High quality proposal in nearly all respects; should be supported if at all possible
• Good: A quality proposal, worthy of support
• Fair: Proposal lacking in one or more critical aspects; key issues need to be addressed
• Poor: Proposal has serious deficiencies

VERY GOOD  as given by the NSF rating system, as shown above, is given as a “High quality proposal in nearly all respects, should be supported if at possible”.

An analysis of the acceptance/rejection rates for different evaluation levels for 2000 for all NSF proposals can be viewed in Figures 7 and 8 here.  There are other proposals that are rejected despite a VERY GOOD rating but most are not. Figure 8 shows that the average funding rating is just slightly above VERY GOOD.  The use of an average, of course, also shows that a single low rating can significantly skew the average, thus giving that reviewer a more significant influence on the decision process than may be justified.

An analysis of just the climate science proposals in the same manner as this 2000 NSF report would be very informative.

Thus out of 10 reviews, 8 recommended that this is a “[h]igh quality proposal in nearly all respects, should be supported if at possible”.

In contacting the NSF Program Manager of the most recent submission (Thomas Torgersen) for an explanation, as told to Lixin Lu by Torgersen, he said: i) our proposal is very fundable, but he has to balance program priorities with limited resources, he has already used up 70% of the available money to fund 50% of the proposals that he considered very fundable (competitive) and that the program was over subscribed.  I followed up with a phone call to him and he told me only about 15% of submitted proposals are funded by this group and that you need better than “Very Good” to be funded. Unless all of his NSF funded proposals received EXCELLENTS, he is not being honest in his communication to me.

The Project Summary of our proposal

“The Earth’s weather and climate is strongly influenced by the properties of the underlying surface. Much of the solar energy that drives the atmosphere first interacts with the land or sea surface. Over land regions this interaction is modulated by surface characteristics such as albedo, aerodynamic roughness length, leaf area index (LAI), etc. As these characteristics change, either from anthropogenic or natural land-cover disturbances, the amount of energy reaching the atmosphere from the land surface, and thus weather and climate, is expected to change. The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System (RAMS) coupled with the Simple Biosphere (SiB) model, SiB-RAMS, will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox (2008). The influence of the land-cover changes on temperature and precipitation will be examined and compared with that expected from CO2-induced climate change (IPCC 2007). The seasonality of the changes in precipitation and temperature due to land-cover change will be explored. Also, the relative importance of each land-cover biophysical parameter to the total simulated change in temperature and precipitation will be assessed.”

We have made a March 29, 2010 request for reconsideration to NSF to Tim Killeen, Assistant Director of Geosciences (he was also formerly a Director of NCAR). The reconsideration process is given on page IV-3  of this NSF report.  We are still waiting a decision on the reconsideration. The reviews for the three submissions, are copied at the end of this post (listed as Appendix A).

My recommendations to the NSF in order to correct are as follows:

• present ALL proposal abstracts, reviews of both accepted and rejected proposals and program managers decision letters (or e-mails) on line for public access
• present the date of submission and final acceptance (or rejection) of the proposal.

APPENDIX A

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Reviews of proposal in 2007

Review #1

Proposal Number:

0735895

Performing Organization:

U of Colorado Boulder

NSF Program:

Climate and Large-scale Dynamics

Principal Investigator:

Strack, John E

Proposal Title:

Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Very Good

REVIEW:

What is the intellectual merit of the proposed activity?

Given the novelty of the historical surface data applied to a regional climate model (RAMS), the proposed numerical experiments will provide a suite of simulations aimed to quantify the impact of historical land cover changes over the eastern U.S. on simulated climate and weather. The authors aim to use these diagnoses to gauge whether land cover changes have played (and could play?) as important a role in climate/weather change as increased greenhouse gas concentrations (based on IPCC estimates). Intellectually speaking, this is a very compelling issue and could provide a salient example of how ‘climate policy’ must not only tackle issues of global emission strategies, but also land use.

What are the broader impacts of the proposed activity?

The proposed study lies within the broader context of understanding to what extent humans have and will modulate weather and climate regimes (in this case through land use/cover change). The results of this study will likely receive a lot of attention from the scientific community-at- large.

Summary Statement

To investigate the potential impacts of (historical) land-cover change on regional (i.e. ‘eastern’ U.S.) climate and weather, the investigators propose a suite of numerical experiments with a proven recipe, which is to replace various surface-vegetation (or lack thereof) parameters according to a recently released historical data set of land cover change, and quantify the response of RAMS to these changes. The investigators will also perform additional experiments to assess the relative impact of each vegetation parameter change on the regional model simulation.

The likely outcome of the proposed study is that they will successfully execute the proposed numerical experiments, conduct very solid analyses, and arrive at some insightful interpretations – but all within the context of the model used and under the scope of substantial uncertainty in many facets of the tools and data used. It was somewhat disappointing that the investigators did not explicitly design any facet of their experiments and/or analyses to address uncertainty of the model and parameters considered. That is not to say that these issues are show-stoppers, but rather the investigators could have articulated some first-order accountability of these uncertainties. In this way, their results would have been better poised to address the scrutiny that will likely come to bear on these results. Nevertheless, the climate change arena continues to increase its attention on regional change assessments and impacts, and the research must respond accordingly. Along these lines, the results obtained from the proposed study will be very compelling, promote lively discussion and consideration amongst the climate community, and pave the way for further studies.

Some specific comments:

The proposed regional model domain for the RAMS experiments is curious. If employed, the investigators will miss substantial portions of the north-central (most of the Great Lakes region) and northeast (i.e. nearly all of New England) portions of the ‘eastern’ United States. Given the results of Figure 1, they will be missing some very notable surface-forcing signals, not to mention substantial population regions. Why have they chosen this domain? Is it possible to expand the RAMS domain slightly so that the entire eastern U.S. (as depicted in Figure 1) is considered? By doing so, the experimental results are much more comprehensive in a geographic sense.

There seems to be only one passage (4th and 5th sentences of Section 4c) in the proposal which alludes to the fact that they would evaluate how capable RAMS is at simulating the present day climate and weather (with the surface data set employed). This would seem to be a fundamental requirement, and deserves much more attention, not only in the context of the proposed experiments and analyses, but also in the proposal. What metrics will the investigators use to determine RAMS’ capability to reproduce observed ‘climate’ and ‘weather’? For the latter, simply looking at precipitation and min/max temperatures does not nearly adequately address the ability of RAMS to simulate the observed ‘weather’. The most likely situation is that RAMS does some things great, and other things not so great, and it’s very important to understand why, particularly in the context of the experiments performed. Further, it seems that RAMS has a strong heritage and therefore likely has been evaluated along these lines already. It’s curious why the proposal did not cite previous studies along these lines to strengthen the case RAMS’ fidelity and credence in its simulated response to land cover/use change.

It would seem that the investigators must run an additional suite of RAMS simulations that also modulate the ambient concentrations of greenhouse gases for the time periods considered (i.e. 1650, 1850, 1920, and 1992). Otherwise, the comparison of the land cover change response of RAMS to the IPCC climate-model response to greenhouse gas concentrations is certainly not a direct (or fair) comparison and any assessment along these lines is substantially weakened (if not invalid) and will likely receive a lot of scrutiny along these lines.

Notwithstanding the aforementioned comments, which were mainly provided in the hopes that the investigators would consider these in recasting their experiments and subsequent analyses, this is a very cost-effective study. The results and interpretations of the proposed study will likely receive a lot of attention, and the bottom line is that it will move the science and understanding of (regional) climate/weather change forward, which in this reviewer’s opinion, is the most that one can ask from studies such as these.

Review #2

Proposal Number:

0735895

Performing Organization:

U of Colorado Boulder

NSF Program:

Climate and Large-scale Dynamics

Principal Investigator:

Strack, John E

Proposal Title:

Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Very Good

REVIEW:

What is the intellectual merit of the proposed activity?

The proposed work aims to investigate the sensitivity of weather and climate in the eastern United States to changes in land cover that have occurred since European settlement, taking advantage of a new historical land cover dataset. More specifically, the investigators, using a regional climate model driven by reanalysis data, hope to quantify the impacts of these changes which are realistic rather than idealized û and compare them to the Intergovernmental Panel on Climate Change projections (IPCC) of future climate. Additional land surface parameter perturbation experiments are proposed to determine the individual roles of the key land surface parameters on the simulated changes in weather and climate resulting from the land cover change experiments.

All of the proposed simulations are to be performed at a 20-km horizontal grid spacing for one year with an additional month for soil moisture and temperature spin up considerations. While the experiments are computationally feasible, they are not nearly long enough to conclusively determine if the resulting changes in climate are similar in magnitude to the IPCC projections, which is the primary hypothesis of the proposed research. In addition, one month is generally not long enough to spin up the soil moisture and soil temperature of a land surface model. The proposed land cover change simulations should be extended to at least 10 years û probably 20 or 30 years û with at least one year of spin up. The land surface parameter perturbation experiments should be extended to 5 years or so. To make this computationally feasible, the horizontal grid-spacing could be increased.

The investigators both seem well-qualified to perform the proposed work. Collectively, they have numerous publications in the area of modeling the impacts land cover changes on climate. In particular, Roger A. Pielke, Sr. is regarded as a leader in the field. Overall, the proposed activity, with longer numerical experiments, should act to greatly advance the knowledge and understanding of the historic role of land cover on climate.

What are the broader impacts of the proposed activity?

The results from the proposed project will benefit society and the research community by increasing the awareness of the role of land cover changes on climate, especially if the changes in climate turn out to be as large as those projected by the IPCC. Collaboration will be developed between the ecologists and atmospheric scientists through data and knowledge exchange. The investigators aim to train students, including those from underrepresented groups, through various training and mentoring activities.

Summary Statement

Overall, the proposed research should result an advancement of the understanding of the role that historical land cover changes have had on present-day climate. Recently, much emphasis has been placed on the role of anthropogenic greenhouse gas emissions on climate, which undoubtedly is important. Little emphasis, however, has been placed on the impacts of anthropogenic land cover changes on climate, which could be as important. The proposed research will help to determine this importance, at least in North America, which could be of particular interest to society.

The main weakness of the proposed research is the short length of the numerical simulations. Nevertheless, coarsening the horizontal grid-spacing would likely compensate the increase in the simulation length keeping the experiments computationally feasible.

In summary, this is a high quality proposal in nearly all respects and should be supported if at all possible.

Proposal Number:

0735895

Performing Organization:

U of Colorado Boulder

NSF Program:

Climate and Large-scale Dynamics

Principal Investigator:

Strack, John E

Proposal Title:

Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Very Good

REVIEW:

What is the intellectual merit of the proposed activity?

The proposal is designed to study the sensitivity of weather and climate to land cover change by using RAMS simulations with changes in the vegetation cover and the associated land surface model parameters for different time frames. The idea is interesting; however the methodology is somewhat short of details. It is not clear which are the criteria to be used to identify the normal, wet and dry years whose forcing is going to be used to run the models.

The proposed initialization of the soil moisture with a spin up of only one month raises some concerns. It might not be enough, and it is well documented that it may play a big role in the results. Also, to study the sensitivity to the LSM parameters on a one at a time basis is not necessarily the best way to proceed, particularly if no interaction is considered. I must acknowledge that somewhere in the text something is mentioned about simultaneous changes in the parameters to be considered as a source of determination of the individual contributions to the overall sensitivity, no specifics are given. At the very least some sort of factorial analysis needs to be included (this approach will be somewhat justified due to the actual computer time required for the simulations). The evaluation methods to be used are not properly described; there is no mention of the time scale to be considered and how the spatial distribution is going to be taken into account. The objectives and the questions posed are of interest and should contribute to the understanding of the land cover change effects on the climate and weather patterns. The investigators have the necessary background to carry out the proposed tasks. It will be of interest if besides communicating the results at different meetings the data resulting from the study will be made available. Nothing is mentioned about that.

What are the broader impacts of the proposed activity?

The proposed work will be of interest by helping to understand the human effect on the land cover change and the effects this has on the climate and weather patterns. This understanding will help in making to general community more aware of the implications. Standard scientific communication is also contemplated in the proposal.

Summary Statement

An interesting proposal, demanding in terms of simulation time; however some details have not been properly thought, e.g. soil moisture initialization, one at a time sensitivity, simulation evaluations.

************************************************************

Here are the reviews from 1st submission by Lixin Lu and RA Pielke

Review #1

Proposal Number: 0840826

Performing Organization: U of Colorado Boulder

NSF Program: Climate and Large-scale Dynamics

Principal Investigator: Lu, Lixin

Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Very Good

REVIEW:

What is the intellectual merit of the proposed activity?

The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. By including runs with different concentrations of CO2, the ultimate goal is to determine how land surface changes and anthropogenic changes in greenhouse gas concentrations interact to contribute to the observed changes in climate. The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined.

Although the proposed research isn’t exactly unique (the authors cite a lengthy list of previous experiments, at least one of which (Baidya- Roy et al., 2003) also used RAMS to study the effect of historical land use changes on the North American climate), the current proposal seeks to build on earlier work by using newer surface data and a newer land surface model, as well as by running a more exhaustive set of simulations to explore many aspects of land surface change on climate.

Pros

The authors have identified a gap in the current understanding of past climate change and its implications for the future. They are quite clear about what they hope to accomplish, what experiments they wish to run, and how they will be carried out (for example, all will use modern reanalysis data to force the mesoscale model, and all will use a 10-15 year spin-up period). Both authors have extensive experience with this type of modeling, and the proposed work can be done using existing, available models, as well as a land surface dataset they have used before. The timeline is reasonable and seems to allow enough time for all the proposed simulations, particularly if each run takes about 24 days to run on a single node. Given the list of well-planned experiments, it is clear that they will have a good idea as to what the relative magnitudes of the effects of land surface parameter changes will be.

Cons

The authors are vague about their use of ensemble modeling. As stated in 4.4, experiments #1-30 will be run for 1 year each. This is not adequate for comparing seasonal means from different simulations. In section 4.5, however, they do talk about ensemble runs they will do, using MLEF to determine the ensemble uncertainty (which is more accurate than simply going by the ensemble variance). They describe how they will produce ensembles for each of the 30 experiments, but then imply later that they won’t be using MLEF. Will they do any ensemble runs? Plus, they do not specify how many members will be in each ensemble, which makes it harder to gauge if the timeline is reasonable.

What are the broader impacts of the proposed activity?

The authors list several impacts, most of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are listed last – ‘increased awareness of how changes in land cover with time can lead to changes in regional weather and climate’ and showing ‘the necessity of accounting for land-cover change in simulations of future climate change’. These are important to the issue of how anthropogenic changes in greenhouse gas concentrations have altered climate in the past and how they will do so in the future. The 2007 IPCC Synthesis Report (IPCC, 2007) indicates that the effect of past land use change has been relatively modest, but also that the level of understanding is ‘medium-to-low’, so there is room for improvement in the understanding of these processes.

IPCC, 2007: Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, Pachauri, R.K and Reisinger, A. (eds.)]. IPCC, Geneva, Switzerland, 104 pp.

Summary Statement

In deciding on the rating, I gave greater weight to the intellectual merit, since this is where the proposal’s greatest strengths lie. The proposal for the most part well-written, and spells out a clear, detailed research project. I was very intrigued by the research and would also be very interested in reading the resultant paper. However, the ensemble modeling that would seem to be so important was only vaguely described. Therefore, this part of the plan needs work.

Review #2

Proposal Number: 0840826

Performing Organization:

U of Colorado Boulder

NSF Program: Climate and Large-scale Dynamics

Principal Investigator: Lu, Lixin

Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Good

REVIEW:

What is the intellectual merit of the proposed activity?

This proposal will attempt to understand the effects of land cover change using a regional model coupled to the terrestrial biosphere model SIB2. The regional model will be driven by reanalysis products. The analysis will investigate the effects in the eastern US through time; 1650-1992.

Effects of land use change is a poorly understood area of research. The authors propose to investigate the effects of land cover changes since 1650 in the eastern US. They also propose to investigate the seasonal effects. They propose to study the sensitivity to various land biophysical parameters as well. The proposal is well written and it will bring interaction of ecologists and atmospheric scientists.

Though there is discussion about carbon cycle, I do not see any proposed analysis on this subject (NPP, plant and soil respiration, biomass, etc.).

There is a major shortcoming in this study. The proposal is weak in the statistical analysis of the significance. The current proposal is to carry out thirty 1-yr experiments. By definition, climate is the statistics of weather. To understand the climate effects, each experiment should be performed at the least over 30 years to get a decent sample. Without performing a proper statistical analysis of the simulations, one can not make meaningful statements about the climate effects of a particular forcing mechanism. I would suggest the authors, for example, to replace their first 15 experiments with the first 4 experiments but each of the 4 experiments run for 30 years. I do not see any value in experiments 13-15 because the effect of CO2 forcing can be computed offline using simple formula relating radiative forcing and temperature change.

The last 15 experiments should be also run for about 30 years each. It this is computationally expensive, then the authors could consider replacing their last 15 experiments with just 5 experiments which explore the sensitivity to the 5 biophysical parameters between for example 1992 and 1920. The authors have compromised the quality of experiments for the sake of quantity.

What are the broader impacts of the proposed activity?

As outlined by the authors, the main impact of this proposal will be the education of a Ph.D. student. The secondary benefit will be the enhancement of the infrastructure for interdisciplinary research between land and atmosphere modelers. However, their design of the experiments will not shed much light into the understanding of the climate effects of land cover change (see the comment on intellectual merit).

Summary Statement

The proposal has the objective of understanding the effects of land cover changes since 1650. It also proposes to study the sensitivity to various land biophysical parameters. The proposal is well written. The funding request appears reasonable.

Since the objective is to identify the impacts on climate, it is vital that the statistical significance is assessed properly and carefully. In climate science, one may simulate large differences between two simulations. However, the differences may not be above the climate noise and hence may not be meaningful. The 1-year experiments proposed in this study will not allow the author properly determine the effects of land cover change on climate. Unless the authors modify their proposal to perform a subset of 30-year simulations, I would not recommend this proposal for funding.

Review #3

Proposal Number: 0840826

Performing Organization: U of Colorado Boulder

NSF Program: Climate and Large-scale Dynamics

Principal Investigator: Lu, Lixin

Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating:

Very Good

REVIEW:

What is the intellectual merit of the proposed activity?

The goal of this project is to evaluate the impact of past land cover change on the climate of the Eastern United States (EUS), using a regional climate model (SiB-RAMS). The proposal comes from two scientists that have been working hard to push forward the understanding of land surface processes and their impact on weather and climate. Furthermore, the proposed work is a logical continuation of a recent study carried out within the group (Strack et al., 2008).

Even thought I wish the choice of EUS as the studied region was explicitly justified, I personally think that this choice is judicious. Indeed, whereas most regions in the world experienced a GHG-driven warming since preindustrial times, EUS on the other hand experienced a slight cooling (IPCC, 2007). Land cover change may have played a role in this observed cooling trend, thus understanding land cover change effect in this region represents an interesting challenge.

Several studies have been investigating the impact of land cover change on the climate of the EUS (Copeland et al., 1996; Baidya Roy et al., 2003; Strack et al., 2008). But I think that in several aspects the proposed work will represent a step forward compared to these previous studies.

First, the proposed simulations will include a more advanced representation of the land surface, accounting for both biogeophysical and biogeochemical processes within vegetation and soil.

Second, the originality of the proposal also comes from the input land cover datasets that will be used in the simulations. This dataset is a newly developed representation of land cover dynamics in the EUS from pre-colonial times to present-day (Steyaert & Knox, 2008). Besides mapping land cover types, this dataset also provides information on biophysical properties associated with each land cover type (albedo, leaf area index, canopy height, rooting depth, surface roughness, etc.). This will enable the study of the sensitivity of climate to change in individual surface parameters, which is necessary to understand the physical mechanisms behind the overall effect of land cover change.

Third, previous mesoscale model simulations of the impact of land cover change on EUS climate have only focused on summer climate. This represents a very limited assessment of land cover change impact. In this project, the PIs propose to examine the impact of land cover change not exclusively on summer climate, but on the full seasonal cycle. From my point of view that represents the main strength of the project.

The proposed experimental design is ambitious, consisting of 20 different 1-year experiments (combining different land cover maps, background CO2, normal, wet or dry years and change in individual parameters). My only frustration about this experimental design is that each experiment has only one single realization (Perhaps would it be reasonable to run a smaller set of experiments but using an ensemble approach?). Previous experiments have shown that the climate signal arising from land cover change is often of small magnitude and sometimes even of opposite signs across different models. Careful statistical analysis is thus required when one wants to extract the signal due to land cover change from the noise. In that respect an ensemble approach would have been very useful.

However, I note that the PIs propose a way to address uncertainties in their simulations using the Maximum Likelihood Ensemble Filter (MLEF). I acknowledge that I have no expertise in this technique, therefore I can hardly tell if it is a relevant choice for the above- mentioned purpose.

What are the broader impacts of the proposed activity?

This project will contribute to the training of a PhD student. It should also benefit the scientific community by disseminating the results through conferences and publications. Furthermore, the PIs propose to make their model results available to the scientific community, which will probably stimulate further analysis by other groups.

By addressing aspects that have not been investigated in previous studies, I believe that this project will advance our understanding of land cover change impact on regional climate.

Ultimately, understanding how past land cover change did affect regional climate should provide guidance for future decisions concerning land management.

Summary Statement

This proposal builds upon strong existing bases and proposes to investigate several original aspects. The outcome from this research can be expected to broaden our understanding of land cover change impacts. That will be valuable for the scientific community and also ultimately for society, because this would help in the decision- making process concerning future land management.

Review #4

Proposal Number: 0840826

Performing Organization: U of Colorado Boulder

NSF Program: Climate and Large-scale Dynamics

Principal Investigator: Lu, Lixin

Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement

Rating: Fair

REVIEW:

What is the intellectual merit of the proposed activity?

The PIs both have a good background in the proposed research. They are highly productive scientists (as evidenced by their publication

records) and have made important contributions to the science. However, I have several concerns about the proposed research.

I do not see evidence of any new, creative, or original concepts. The research is simply an exploratory expedition using a particular model. The PIs plan 30 1-year simulations with various changes in the land surface or atmospheric CO2 concentration to see what happens. They plan to change land cover for various points in time (1992, 1920, 1850, 1650), change CO2 concentration, and then individually change specific land surface parameters (e.g., albedo, roughness, LAI, rooting depth). There are no new ideas that will greatly alter our understanding of land-atmosphere interactions.

The robustness of these results is not clear, or even considered in the proposed work plan. How much will the results depend on specific model formulations? The sensitivity to soil moisture is very important. This is discussed in little detail, except to say that simulations will be performed for normal, wet, and dry years. There is no discussion of ensemble simulations. Little thought is given to model initialization and how this initialization will effect the simulations (1-year in length). Soil moisture will be initialized from offline simulations of SiB2 driven by reanalysis data, but how sensitive will the 1-year simulations be to this?

The budget is very large . a total of $657,000 over three years . for the amount of results that will be gained. It supports 7 months of the PIs’ salary, 2 months of a professional research assistant, and 1 Ph.D. student.

What are the broader impacts of the proposed activity?

The proposed research will train 1 PhD student.

Summary Statement

The PIs have a strong research record, but this proposal has been put together in a very cursory manner. New, creative, or original concepts are lacking.

—————————

Most Recent Reviews

Date: Tue, 2 Feb 2010 12:35:44 -0500

From: “Torgersen, Thomas”

To: lixin@cira.colostate.edu, pielkesr@cires.colorado.edu

Subject: Collaborative Research: Sensitivity of weather and climate in the

Eastern United States to

Drs. Lu and Pielke,

I attach the reviews and comments on your recent proposal. At this point, I would admit that the budget is tight and this is falling on the “not enough funds” side of things. However, I have one more idea that I will pursue however much a stretch it is…

Nonetheless, I wanted you to have the reviews and comments available if you were planning on submitting a (category 3) WSC proposal.

PROGRAM OFFICER COMMENTS

0943712 Lu and Pielke

Collaborative Research: Sensitivity of weather and climate in the Eastern United States to historical land-cover changes since European Settlement

$534,372 Lixin Lu Colorado State University

176,532 Roger A Pielke, SR University of Colorado at Boulder

The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System

(RAMS) coupled with the Simple Biosphere (SiB) model will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox

(2008) which uses 36 land-cover categories to describe the landscape for 4 time slices, 1650, 1850, 1920, and 1992 in the United States east of 97ºW. This project will rework Strack et al 2008 (land cover caused 0.3-0.4C temp

increase) with a better model. The observed land-cover and land-use changes and the climate system feedbacks associated with these changes, are expected to have important ramifications for the eastern U.S. weather and climate, as well as energy, water, and carbon budgets. Understanding the associated consequences requires a modeling system capable of accounting for the relatively high-resolution land cover and vegetation biophysical parameter changes, and the related changes in energy, moisture, momentum, and carbon fluxes which is within the capability of RAMS/SiB and is capable of simulating critical atmosphere, ocean, and land-surface interactions

AD HOC REVIEWER COMMENTS (direct quotes)

Reviewer 1 GOOD

…the methods, model, and scenario design seem well-suited to addressing the questions… I would not rate the proposal as transformative (most proposals are not), but it does represent a solid next step in the research this group has been doing…. This is a solid scientific proposal on a very relevant topic, a logical next step in the research this group and others have been doing using an interesting new data set on historical land use, and a very competent research group. The budget seems a bit high, the broader impacts are routine at best.

Reviewer 2 VERY GOOD

I do have some concern with whether the proposal is transformative. I do not believe that idea or tools proposed are transformative because the proposal is simply to run a regional weather model for different land use parameterizations. However, the results from this work do have the potential to be transformative if, as the authors hypothesize, the changes in temperature and precipitation due to land use change are found to be comparable to those expected from the IPCC multi-decadal global model simulations… The primary weakness that I see is related to the broader impacts

Reviewer 3 R

it is not clear if the nutrient effects on LAI are included in the analysis…

Reviewer 4 VERY GOOD

The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. … The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined… The authors list several impacts, some of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are providing a baseline to determine how future land-use chance can alter climate, and to ‘increase our understanding of the mechanisms by which land-cover change affects the atmosphere’

PANEL COMMENTS (direct quotes)

The stated goals are to understand the mechanisms by which land cover affects climate systems and to articulate the need for better land-cover representation in climate models. While the later is well known, the former has not been well articulated in the proposal… there is no effort to integrate the nutrient cycle. This was seen as weakness. Another weakness identified was the use of representative years for simulations to make climatological inferences. No effort is made to articulate how “representative” are the representative years… most parameters are fairly simple to obtain.

PROGRAM OFFICER ANALYSIS:

The project represents the next logical step in the analysis of land-use and climate variability through time over the eastern US. The hope would be that the model would “define” the role of land-use change and the residual might be indicative of climate change impacts. While this comparison can be made and will yield some results, the concern is that the results/contribution would be incremental; this remains high risk because it cannot be determined IF the two impacts can be separated. If the residual is small, does that mean climate change is small impact or that the model has wide error for land-use change? Given that this project merges existing models and datasets, it would be more beneficial to implement this effort on the path towards something more substantial e.g. parameter sensitivity assessment of RAMS/SiB and/or sensitivity to patch size of land-use or runoff variability as a function of land use and climate change and wet vs. dry. In its present design, it certainly does represent the next logical step but at a cost that it has not adequately justified in terms of the product to be produced. The results could be high payoff but at the present formulation, the risk is also high. The reviews and the panel analyses are generally weaker than fundable. I would urge the PIs to stretch themselves further with each proposal.

Thomas Torgersen 2 February 2010

Program Officer, Hydrologic Sciences

FULL REVIEWS AND PANEL SUMMARIES FOLLOW

Reviewer 1

Rating: Good

Review:

What is the intellectual merit of the proposed activity?

This research aims to use a new research product (historical land cover for eastern US in 1650, 1850, 1920, 1990, including biophysical parameter reconstruction (from Steyaert and Knox 2008); in the SiB-RAMS model to look at the impact of land cover change on annual and seasonal weather and climate. They pose an interesting question û were these changes (magnitude yet to be determined) of the same size (or larger) than projected 21st Century climate change? In 2008 Strack, Pielke, Steyaert and Knox published what must be a preliminary study for this proposed work: ‘sensitivity of June near surface temperatures and precipitation in the Eastern United States to historical land cover change since European settlement’ (it would have been good to see an exciting result from this study). This proposed project will expend this to look at variation through the year (seasonality) and also carbon balances in addition to weather variables.

The objectives are clearly stated and the methods, model, and scenario design seem well-suited to addressing the questions.

The group is highly qualified to do the research; they have extensive experience with the SiB-RAMS model (and other RAMS and RAMS/other models), have a solid (or better) record of publishing their results. Pielke has been a leader in this field for decades.

I would not rate the proposal as transformative (most proposals are not), but it does represent a solid next step in the research this group has been doing.

It is not clear exactly how much work there is and if it justifies 6 months per year of PI Lu’s time, a full-time graduate student, plus 1 month of Pielke’s time (reasonable) and 2 months for a Pielke assistant. There is very little model development proposed; the 30 simulation studies seem fairly straight-forward, though computationally demanding, to implement. These will require some attention and time to make sure the model keeps running and the results are saved, etc. etc. There will be a large amount of model output to process and analyze, presumably primarily by the graduate student, with supervision and assistance from Lu and Pielke.

The computing requirements for this project seem large – one 365-day simulation run is 24 wall-clock CPU days (I’m not exactly sure what this means?), so 30 scenarios = 720 wall-clock CPU days = 2 yrs (if nothing goes wrong). What if computing time not awarded by NCAR (maybe this is routine for these researchers)? Do they have sufficient computing available in that case?

What are the broader impacts of the proposed activity?

The broader impacts of this study are

(1) Scientific results on a topic (impacts of land use on climate) that is of interest and relevance to society and to a larger community scientific community. These results will be presented at scientific meetings and published in the scientific literature, and available on-line, and can be expected to contribute relevant and high-qualify science to the field of climate and global change.

(2) Education and training of a PhD student, in an important, interdisciplinary scientific field. The proposal also states that ‘other students and researchers, including those from underrepresented groups, will be trained in the use of the model that is applied in our study,’ but it is not explained how this will occur.

Summary Statement

This is a solid scientific proposal on a very relevant topic, a logical next step in the research this group and others have been doing using an interesting new data set on historical land use, and a very competent research group. The budget seems a bit high, the broader impacts are routine at best.

Reviewer 2

Rating: Very Good

Review:

What is the intellectual merit of the proposed activity?

This is a very well conceived, written, and organized proposal. The project team seems to be qualified to carry out the proposed work given their past experience in atmospheric modeling. Understanding impacts of land use change on weather and climate is an important science question, and the land use change dataset available to the researchers will allow for original work on quantifying the impacts of land use change from 1650 to present day.

I do not have experience with the models mentioned in the proposal and cannot speak to their applicability to the stated research hypothesis. To my mind, the authors have demonstrated that the models are appropriate for addressing the research objectives outlined in the proposal, although I would defer to an expert in climate modeling on this point.

I do have some concern with whether the proposal is transformative. I do not believe that idea or tools proposed are transformative because the proposal is simply to run a regional weather model for different land use parameterizations. However, the results from this work do have the potential to be transformative if, as the authors hypothesize, the changes in temperature and precipitation due to land use change are found to be comparable to those expected from the IPCC multi-decadal global model simulations.

What are the broader impacts of the proposed activity?

The proposed work will fund a PhD student who will be exposed to interdisciplinary studies including the use of cutting-edge models and observations at one of the nation’s best earth system science programs. The results will be disseminated through journal publications, conference presentations, and the project team’s own webpage. A potential impact to the larger society is improved understanding of how regional land use changes can alter climate systems.

Very little is mentioned about broadening the participation of underrepresented groups, other than mentioning that underrepresented groups will be included within the educational goals (with no specifics on how this will be done). The proposal does not address how the work will enhance the infrastructure for research and education.

Summary Statement

This is a well formulated and presented proposal by a seemingly qualified team that has the potential to offer important scientific understanding relating to the impact of land use change on climate systems. The primary weakness that I see is related to the broader impacts and the lack of a clear plan for involving underrepresented groups, for promoting teaching, training, and learning, and for enhancing research and teaching infrastructure.

Reviewer 3

Rating: R

Review:

What is the intellectual merit of the proposed activity?

This is a project to examine the effects of historical land-cover changes in the eastern US on land-atmosphere interactions and regional climate. They will assess their results relative to those from existing climate models from the IPCC, compare the magnitudes of changing land-cover effects among seasons, and evaluate the relative contributions of various biophysical factors (albedo, LAI, roughness, rooting depth, soil water saturation) on their predicted changes in climate and weather.

The project is ambitious and important; examining the effects of past land use change on local climate is a prerequisite for predicting and managing the effects of future land use change. There seems to be a lot of work proposed, but this is the team with the expertise to do it and their management plan indicates a well thought out approach and schedule.

My only concern is the carbon-water focus of the study. In a region so thoroughly worked over by agriculture, especially the initialization and then abandonment of agriculture, the effects of nitrogen and phosphorus are likely to be very important. Some of these effects might be accounted for through LAI, but it is not clear if the nutrient effects on LAI are included in the analysis. In any case, there are other effects of nutrients on plant and soil function other than changes in LAI, e.g., fertilization can enhance microbial respiration, increase plant production per unit leaf area, change rooting density and depth distribution, etc.

What are the broader impacts of the proposed activity?

Broader impacts:

It is clear that this type of work is vital if society is going to understand the impacts of climate change and adapt to it. The requisite mention of students and underrepresented groups is mentioned but not elaborated on (although I think one of the PI’s qualifies in the latter category). There is a request for support for 3years support for a graduate student(s).

Summary Statement:

See above.

Reviewer 4

Rating: Very Good

Review:

What is the intellectual merit of the proposed activity?

The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. By including runs with different concentrations of CO2, the ultimate goal is to determine how land surface changes and anthropogenic changes in greenhouse gas concentrations interact to contribute to the observed changes in climate. The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined.

Although the proposed research isn’t exactly unique (the authors cite a lengthy list of previous experiments, at least one of which (Baidya-Roy et al., 2003) also used RAMS to study the effect of historical land use changes on the North American climate), the current proposal seeks to build on earlier work by using newer surface data and a newer land surface model, as well as by running a more exhaustive set of simulations to explore many aspects of land surface change on climate.

Pros

The authors have identified a gap in the current understanding of past climate change and its implications for the future. They are quite clear about what they hope to accomplish, what experiments they wish to run, and how they will be carried out (for example, all will use modern reanalysis data to force the mesoscale model, and all will use a 10-15 year spin-up period). Both authors have experience with this type of modeling, and the proposed work can be done using existing, available models, as well as an existing land surface dataset. The timeline is reasonable and seems to allow enough time for all the proposed simulations, particularly if each run takes about 24 days to run on a single node. Given the list of well-planned experiments, it is clear that they will have a good idea as to what the relative magnitudes of the effects of land surface parameter changes will be.

Cons

In the original version of this proposal, the authors were vague about their use of ensemble modeling. Now, they have said more about it, mentioning that they will run each experiment 10 times. They are also more definite about their use of MLEF to determine the model uncertainty based on the ensemble results. The only problems I foresee are that they will get the uncertainty in the remote sensing products ‘through a literature review’, which assumes that the needed data is indeed published and available. Have they checked this? Also, the MLEF process is rather involved, and I think a more detailed description of processes like the Monte Carlo analysis at the flux tower is called for.

What are the broader impacts of the proposed activity?

The authors list several impacts, some of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are providing a baseline to determine how future land-use chance can alter climate, and to ‘increase our understanding of the mechanisms by which land-cover change affects the atmosphere’. These are important to the issue of how anthropogenic changes in greenhouse gas concentrations have altered climate in the past and how they will do so in the future. The 2007 IPCC Synthesis Report (IPCC, 2007) indicates that the effect of past land use change has been relatively modest, but also that the level of understanding is ‘medium-to-low’, so there is room for improvement in the understanding of these processes.

IPCC, 2007: Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, Pachauri, R.K and Reisinger, A. (eds.)]. IPCC, Geneva, Switzerland, 104 pp.

Summary Statement

In deciding on the rating, I gave greater weight to the intellectual merit, since this is where the proposal’s greatest strengths lie. The proposal for the most part well-written, and spells out a clear, detailed research project. I was very intrigued by the research and would also be very interested in reading the resultant paper.

Panel Summary

In this proposal the PIs wish to take advantage of a new biophysical parameter dataset produced by Steyaert and Knox to perform coupled land-atmosphere model simulations to examine how the coupling between land and atmosphere has changed over a period of time. In particular they will examine the effect on temperature and precipitation in the Eastern US. The use of the dataset for the stated goal was seen as compelling. The strategy is to perform simulation over climatologically representative years. The stated goals are to understand the mechanisms by which land cover affects climate systems and to articulate the need for better land-cover representation in climate models. While the later is well known, the former has not been well articulated in the proposal. Although the subject matter is certainly timely and results could be highly significant if input data are sound, there does not seem to be a significant technique development component.

The reviews are generally supportive of the proposal. However, over such a long period of time, 1600s to 2000, the nutrient cycle undergoes significant changes, particularly in response to the land cover change. While the carbon cycle is incorporated in the description, there is no effort to integrate the nutrient cycle. This was seen as weakness. Another weakness identified was the use of representative years for simulations to make climatological inferences. No effort is made to articulate how “representative” are the representative years in the context of the interannual variability prevailing for the time period of study and the results can be used to make broad generalizations. If Steyaert and Knox maps and parameters are dependable, the proposed research could provide significant insight into land-atmosphere interactions and carbon cycling. However, most parameters are fairly simple to obtain. Does simply running the model justify the budget? There isn’t much detail on how under represented groups will be served.