This post is titled “Is The NSF Funding Process Working Correctly?”
ABSTRACT
The short answer to this question is NO. I recommend the following changes in the NSF review procedures to improve the process:
TEXT OF MY POST
From my experiences in recent years, in terms of climate science issues, the answer is NO. Indeed, while I will discuss just one (rejected) proposal below, I also have a proposal still under consideration that was submitted on May 13 2009 (an NSF-accomplishment based proposal) but only sent out for review in late March 2010! I will update on my weblog the disposition of that proposal when it is finally decided on).
My experience with the NSF funding process of climate science in recent years fits with Dick Lindzen’s summary (see)
“In brief, we have the new paradigm where…. government largely determines the nature of scientific activity…..”
Perhaps my experience is unique, although from discussions with colleagues who have been visible in questioning the IPCC perspective of climate change, there are others who have had similar treatment in the NSF funding process.
In the following text, I document my experiences with a climate science proposal (since we kept the e-mail documentation, the information is readily available).
I will summarize the situation first, and provide more detail later in the post as an Appendix.
NSF Proposal: “Collaborative Research: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes since European Settlement” [with Lixin Lu of Colorado State University and myself at the University of Colorado at Boulder]
This proposal, modified each time in response to the reviews with a change of PIs after the first submission [since funding was lost for the PI of the first submission and he had to leave my research group as a result (John Strack)] has been rejected three times. The evaluations each time were
1. Submission 1 3 VERY GOODs
2. Submission 2 2 VERY GOODs; 1 GOOD; 1 FAIR
3. Submission 3 3 VERY GOODs
The NSF Review Rating Scale (slide 17) is
• Excellent: Outstanding proposal in all respects; deserves highest priority for support
• Very Good: High quality proposal in nearly all respects; should be supported if at all possible
• Good: A quality proposal, worthy of support
• Fair: Proposal lacking in one or more critical aspects; key issues need to be addressed
• Poor: Proposal has serious deficiencies
VERY GOOD as given by the NSF rating system, as shown above, is given as a “High quality proposal in nearly all respects, should be supported if at possible”.
An analysis of the acceptance/rejection rates for different evaluation levels for 2000 for all NSF proposals can be viewed in Figures 7 and 8 here. There are other proposals that are rejected despite a VERY GOOD rating but most are not. Figure 8 shows that the average funding rating is just slightly above VERY GOOD. The use of an average, of course, also shows that a single low rating can significantly skew the average, thus giving that reviewer a more significant influence on the decision process than may be justified.
An analysis of just the climate science proposals in the same manner as this 2000 NSF report would be very informative.
Thus out of 10 reviews, 8 recommended that this is a “[h]igh quality proposal in nearly all respects, should be supported if at possible”.
In contacting the NSF Program Manager of the most recent submission (Thomas Torgersen) for an explanation, as told to Lixin Lu by Torgersen, he said: i) our proposal is very fundable, but he has to balance program priorities with limited resources, he has already used up 70% of the available money to fund 50% of the proposals that he considered very fundable (competitive) and that the program was over subscribed. I followed up with a phone call to him and he told me only about 15% of submitted proposals are funded by this group and that you need better than “Very Good” to be funded. Unless all of his NSF funded proposals received EXCELLENTS, he is not being honest in his communication to me.
The Project Summary of our proposal
“The Earth’s weather and climate is strongly influenced by the properties of the underlying surface. Much of the solar energy that drives the atmosphere first interacts with the land or sea surface. Over land regions this interaction is modulated by surface characteristics such as albedo, aerodynamic roughness length, leaf area index (LAI), etc. As these characteristics change, either from anthropogenic or natural land-cover disturbances, the amount of energy reaching the atmosphere from the land surface, and thus weather and climate, is expected to change. The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System (RAMS) coupled with the Simple Biosphere (SiB) model, SiB-RAMS, will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox (2008). The influence of the land-cover changes on temperature and precipitation will be examined and compared with that expected from CO2-induced climate change (IPCC 2007). The seasonality of the changes in precipitation and temperature due to land-cover change will be explored. Also, the relative importance of each land-cover biophysical parameter to the total simulated change in temperature and precipitation will be assessed.”
We have made a March 29, 2010 request for reconsideration to NSF to Tim Killeen, Assistant Director of Geosciences (he was also formerly a Director of NCAR). The reconsideration process is given on page IV-3 of this NSF report. We are still waiting a decision on the reconsideration. The reviews for the three submissions, are copied at the end of this post (listed as Appendix A).
My recommendations to the NSF in order to correct are as follows:
- present ALL proposal abstracts, reviews of both accepted and rejected proposals and program managers decision letters (or e-mails) on line for public access
- present the date of submission and final acceptance (or rejection) of the proposal.
APPENDIX A
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Reviews of proposal in 2007
Review #1
Proposal Number:
0735895
Performing Organization:
U of Colorado Boulder
NSF Program:
Climate and Large-scale Dynamics
Principal Investigator:
Strack, John E
Proposal Title:
Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
Given the novelty of the historical surface data applied to a regional climate model (RAMS), the proposed numerical experiments will provide a suite of simulations aimed to quantify the impact of historical land cover changes over the eastern U.S. on simulated climate and weather. The authors aim to use these diagnoses to gauge whether land cover changes have played (and could play?) as important a role in climate/weather change as increased greenhouse gas concentrations (based on IPCC estimates). Intellectually speaking, this is a very compelling issue and could provide a salient example of how ‘climate policy’ must not only tackle issues of global emission strategies, but also land use.
What are the broader impacts of the proposed activity?
The proposed study lies within the broader context of understanding to what extent humans have and will modulate weather and climate regimes (in this case through land use/cover change). The results of this study will likely receive a lot of attention from the scientific community-at- large.
Summary Statement
To investigate the potential impacts of (historical) land-cover change on regional (i.e. ‘eastern’ U.S.) climate and weather, the investigators propose a suite of numerical experiments with a proven recipe, which is to replace various surface-vegetation (or lack thereof) parameters according to a recently released historical data set of land cover change, and quantify the response of RAMS to these changes. The investigators will also perform additional experiments to assess the relative impact of each vegetation parameter change on the regional model simulation.
The likely outcome of the proposed study is that they will successfully execute the proposed numerical experiments, conduct very solid analyses, and arrive at some insightful interpretations – but all within the context of the model used and under the scope of substantial uncertainty in many facets of the tools and data used. It was somewhat disappointing that the investigators did not explicitly design any facet of their experiments and/or analyses to address uncertainty of the model and parameters considered. That is not to say that these issues are show-stoppers, but rather the investigators could have articulated some first-order accountability of these uncertainties. In this way, their results would have been better poised to address the scrutiny that will likely come to bear on these results. Nevertheless, the climate change arena continues to increase its attention on regional change assessments and impacts, and the research must respond accordingly. Along these lines, the results obtained from the proposed study will be very compelling, promote lively discussion and consideration amongst the climate community, and pave the way for further studies.
Some specific comments:
The proposed regional model domain for the RAMS experiments is curious. If employed, the investigators will miss substantial portions of the north-central (most of the Great Lakes region) and northeast (i.e. nearly all of New England) portions of the ‘eastern’ United States. Given the results of Figure 1, they will be missing some very notable surface-forcing signals, not to mention substantial population regions. Why have they chosen this domain? Is it possible to expand the RAMS domain slightly so that the entire eastern U.S. (as depicted in Figure 1) is considered? By doing so, the experimental results are much more comprehensive in a geographic sense.
There seems to be only one passage (4th and 5th sentences of Section 4c) in the proposal which alludes to the fact that they would evaluate how capable RAMS is at simulating the present day climate and weather (with the surface data set employed). This would seem to be a fundamental requirement, and deserves much more attention, not only in the context of the proposed experiments and analyses, but also in the proposal. What metrics will the investigators use to determine RAMS’ capability to reproduce observed ‘climate’ and ‘weather’? For the latter, simply looking at precipitation and min/max temperatures does not nearly adequately address the ability of RAMS to simulate the observed ‘weather’. The most likely situation is that RAMS does some things great, and other things not so great, and it’s very important to understand why, particularly in the context of the experiments performed. Further, it seems that RAMS has a strong heritage and therefore likely has been evaluated along these lines already. It’s curious why the proposal did not cite previous studies along these lines to strengthen the case RAMS’ fidelity and credence in its simulated response to land cover/use change.
It would seem that the investigators must run an additional suite of RAMS simulations that also modulate the ambient concentrations of greenhouse gases for the time periods considered (i.e. 1650, 1850, 1920, and 1992). Otherwise, the comparison of the land cover change response of RAMS to the IPCC climate-model response to greenhouse gas concentrations is certainly not a direct (or fair) comparison and any assessment along these lines is substantially weakened (if not invalid) and will likely receive a lot of scrutiny along these lines.
Notwithstanding the aforementioned comments, which were mainly provided in the hopes that the investigators would consider these in recasting their experiments and subsequent analyses, this is a very cost-effective study. The results and interpretations of the proposed study will likely receive a lot of attention, and the bottom line is that it will move the science and understanding of (regional) climate/weather change forward, which in this reviewer’s opinion, is the most that one can ask from studies such as these.
Review #2
Proposal Number:
0735895
Performing Organization:
U of Colorado Boulder
NSF Program:
Climate and Large-scale Dynamics
Principal Investigator:
Strack, John E
Proposal Title:
Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
The proposed work aims to investigate the sensitivity of weather and climate in the eastern United States to changes in land cover that have occurred since European settlement, taking advantage of a new historical land cover dataset. More specifically, the investigators, using a regional climate model driven by reanalysis data, hope to quantify the impacts of these changes which are realistic rather than idealized û and compare them to the Intergovernmental Panel on Climate Change projections (IPCC) of future climate. Additional land surface parameter perturbation experiments are proposed to determine the individual roles of the key land surface parameters on the simulated changes in weather and climate resulting from the land cover change experiments.
All of the proposed simulations are to be performed at a 20-km horizontal grid spacing for one year with an additional month for soil moisture and temperature spin up considerations. While the experiments are computationally feasible, they are not nearly long enough to conclusively determine if the resulting changes in climate are similar in magnitude to the IPCC projections, which is the primary hypothesis of the proposed research. In addition, one month is generally not long enough to spin up the soil moisture and soil temperature of a land surface model. The proposed land cover change simulations should be extended to at least 10 years û probably 20 or 30 years û with at least one year of spin up. The land surface parameter perturbation experiments should be extended to 5 years or so. To make this computationally feasible, the horizontal grid-spacing could be increased.
The investigators both seem well-qualified to perform the proposed work. Collectively, they have numerous publications in the area of modeling the impacts land cover changes on climate. In particular, Roger A. Pielke, Sr. is regarded as a leader in the field. Overall, the proposed activity, with longer numerical experiments, should act to greatly advance the knowledge and understanding of the historic role of land cover on climate.
What are the broader impacts of the proposed activity?
The results from the proposed project will benefit society and the research community by increasing the awareness of the role of land cover changes on climate, especially if the changes in climate turn out to be as large as those projected by the IPCC. Collaboration will be developed between the ecologists and atmospheric scientists through data and knowledge exchange. The investigators aim to train students, including those from underrepresented groups, through various training and mentoring activities.
Summary Statement
Overall, the proposed research should result an advancement of the understanding of the role that historical land cover changes have had on present-day climate. Recently, much emphasis has been placed on the role of anthropogenic greenhouse gas emissions on climate, which undoubtedly is important. Little emphasis, however, has been placed on the impacts of anthropogenic land cover changes on climate, which could be as important. The proposed research will help to determine this importance, at least in North America, which could be of particular interest to society.
The main weakness of the proposed research is the short length of the numerical simulations. Nevertheless, coarsening the horizontal grid-spacing would likely compensate the increase in the simulation length keeping the experiments computationally feasible.
In summary, this is a high quality proposal in nearly all respects and should be supported if at all possible.
Proposal Number:
0735895
Performing Organization:
U of Colorado Boulder
NSF Program:
Climate and Large-scale Dynamics
Principal Investigator:
Strack, John E
Proposal Title:
Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
The proposal is designed to study the sensitivity of weather and climate to land cover change by using RAMS simulations with changes in the vegetation cover and the associated land surface model parameters for different time frames. The idea is interesting; however the methodology is somewhat short of details. It is not clear which are the criteria to be used to identify the normal, wet and dry years whose forcing is going to be used to run the models.
The proposed initialization of the soil moisture with a spin up of only one month raises some concerns. It might not be enough, and it is well documented that it may play a big role in the results. Also, to study the sensitivity to the LSM parameters on a one at a time basis is not necessarily the best way to proceed, particularly if no interaction is considered. I must acknowledge that somewhere in the text something is mentioned about simultaneous changes in the parameters to be considered as a source of determination of the individual contributions to the overall sensitivity, no specifics are given. At the very least some sort of factorial analysis needs to be included (this approach will be somewhat justified due to the actual computer time required for the simulations). The evaluation methods to be used are not properly described; there is no mention of the time scale to be considered and how the spatial distribution is going to be taken into account. The objectives and the questions posed are of interest and should contribute to the understanding of the land cover change effects on the climate and weather patterns. The investigators have the necessary background to carry out the proposed tasks. It will be of interest if besides communicating the results at different meetings the data resulting from the study will be made available. Nothing is mentioned about that.
What are the broader impacts of the proposed activity?
The proposed work will be of interest by helping to understand the human effect on the land cover change and the effects this has on the climate and weather patterns. This understanding will help in making to general community more aware of the implications. Standard scientific communication is also contemplated in the proposal.
Summary Statement
An interesting proposal, demanding in terms of simulation time; however some details have not been properly thought, e.g. soil moisture initialization, one at a time sensitivity, simulation evaluations.
************************************************************
Here are the reviews from 1st submission by Lixin Lu and RA Pielke
Review #1
Proposal Number: 0840826
Performing Organization: U of Colorado Boulder
NSF Program: Climate and Large-scale Dynamics
Principal Investigator: Lu, Lixin
Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. By including runs with different concentrations of CO2, the ultimate goal is to determine how land surface changes and anthropogenic changes in greenhouse gas concentrations interact to contribute to the observed changes in climate. The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined.
Although the proposed research isn’t exactly unique (the authors cite a lengthy list of previous experiments, at least one of which (Baidya- Roy et al., 2003) also used RAMS to study the effect of historical land use changes on the North American climate), the current proposal seeks to build on earlier work by using newer surface data and a newer land surface model, as well as by running a more exhaustive set of simulations to explore many aspects of land surface change on climate.
Pros
The authors have identified a gap in the current understanding of past climate change and its implications for the future. They are quite clear about what they hope to accomplish, what experiments they wish to run, and how they will be carried out (for example, all will use modern reanalysis data to force the mesoscale model, and all will use a 10-15 year spin-up period). Both authors have extensive experience with this type of modeling, and the proposed work can be done using existing, available models, as well as a land surface dataset they have used before. The timeline is reasonable and seems to allow enough time for all the proposed simulations, particularly if each run takes about 24 days to run on a single node. Given the list of well-planned experiments, it is clear that they will have a good idea as to what the relative magnitudes of the effects of land surface parameter changes will be.
Cons
The authors are vague about their use of ensemble modeling. As stated in 4.4, experiments #1-30 will be run for 1 year each. This is not adequate for comparing seasonal means from different simulations. In section 4.5, however, they do talk about ensemble runs they will do, using MLEF to determine the ensemble uncertainty (which is more accurate than simply going by the ensemble variance). They describe how they will produce ensembles for each of the 30 experiments, but then imply later that they won’t be using MLEF. Will they do any ensemble runs? Plus, they do not specify how many members will be in each ensemble, which makes it harder to gauge if the timeline is reasonable.
What are the broader impacts of the proposed activity?
The authors list several impacts, most of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are listed last – ‘increased awareness of how changes in land cover with time can lead to changes in regional weather and climate’ and showing ‘the necessity of accounting for land-cover change in simulations of future climate change’. These are important to the issue of how anthropogenic changes in greenhouse gas concentrations have altered climate in the past and how they will do so in the future. The 2007 IPCC Synthesis Report (IPCC, 2007) indicates that the effect of past land use change has been relatively modest, but also that the level of understanding is ‘medium-to-low’, so there is room for improvement in the understanding of these processes.
IPCC, 2007: Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, Pachauri, R.K and Reisinger, A. (eds.)]. IPCC, Geneva, Switzerland, 104 pp.
Summary Statement
In deciding on the rating, I gave greater weight to the intellectual merit, since this is where the proposal’s greatest strengths lie. The proposal for the most part well-written, and spells out a clear, detailed research project. I was very intrigued by the research and would also be very interested in reading the resultant paper. However, the ensemble modeling that would seem to be so important was only vaguely described. Therefore, this part of the plan needs work.
Review #2
Proposal Number: 0840826
Performing Organization:
U of Colorado Boulder
NSF Program: Climate and Large-scale Dynamics
Principal Investigator: Lu, Lixin
Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Good
REVIEW:
What is the intellectual merit of the proposed activity?
This proposal will attempt to understand the effects of land cover change using a regional model coupled to the terrestrial biosphere model SIB2. The regional model will be driven by reanalysis products. The analysis will investigate the effects in the eastern US through time; 1650-1992.
Effects of land use change is a poorly understood area of research. The authors propose to investigate the effects of land cover changes since 1650 in the eastern US. They also propose to investigate the seasonal effects. They propose to study the sensitivity to various land biophysical parameters as well. The proposal is well written and it will bring interaction of ecologists and atmospheric scientists.
Though there is discussion about carbon cycle, I do not see any proposed analysis on this subject (NPP, plant and soil respiration, biomass, etc.).
There is a major shortcoming in this study. The proposal is weak in the statistical analysis of the significance. The current proposal is to carry out thirty 1-yr experiments. By definition, climate is the statistics of weather. To understand the climate effects, each experiment should be performed at the least over 30 years to get a decent sample. Without performing a proper statistical analysis of the simulations, one can not make meaningful statements about the climate effects of a particular forcing mechanism. I would suggest the authors, for example, to replace their first 15 experiments with the first 4 experiments but each of the 4 experiments run for 30 years. I do not see any value in experiments 13-15 because the effect of CO2 forcing can be computed offline using simple formula relating radiative forcing and temperature change.
The last 15 experiments should be also run for about 30 years each. It this is computationally expensive, then the authors could consider replacing their last 15 experiments with just 5 experiments which explore the sensitivity to the 5 biophysical parameters between for example 1992 and 1920. The authors have compromised the quality of experiments for the sake of quantity.
What are the broader impacts of the proposed activity?
As outlined by the authors, the main impact of this proposal will be the education of a Ph.D. student. The secondary benefit will be the enhancement of the infrastructure for interdisciplinary research between land and atmosphere modelers. However, their design of the experiments will not shed much light into the understanding of the climate effects of land cover change (see the comment on intellectual merit).
Summary Statement
The proposal has the objective of understanding the effects of land cover changes since 1650. It also proposes to study the sensitivity to various land biophysical parameters. The proposal is well written. The funding request appears reasonable.
Since the objective is to identify the impacts on climate, it is vital that the statistical significance is assessed properly and carefully. In climate science, one may simulate large differences between two simulations. However, the differences may not be above the climate noise and hence may not be meaningful. The 1-year experiments proposed in this study will not allow the author properly determine the effects of land cover change on climate. Unless the authors modify their proposal to perform a subset of 30-year simulations, I would not recommend this proposal for funding.
Review #3
Proposal Number: 0840826
Performing Organization: U of Colorado Boulder
NSF Program: Climate and Large-scale Dynamics
Principal Investigator: Lu, Lixin
Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating:
Very Good
REVIEW:
What is the intellectual merit of the proposed activity?
The goal of this project is to evaluate the impact of past land cover change on the climate of the Eastern United States (EUS), using a regional climate model (SiB-RAMS). The proposal comes from two scientists that have been working hard to push forward the understanding of land surface processes and their impact on weather and climate. Furthermore, the proposed work is a logical continuation of a recent study carried out within the group (Strack et al., 2008).
Even thought I wish the choice of EUS as the studied region was explicitly justified, I personally think that this choice is judicious. Indeed, whereas most regions in the world experienced a GHG-driven warming since preindustrial times, EUS on the other hand experienced a slight cooling (IPCC, 2007). Land cover change may have played a role in this observed cooling trend, thus understanding land cover change effect in this region represents an interesting challenge.
Several studies have been investigating the impact of land cover change on the climate of the EUS (Copeland et al., 1996; Baidya Roy et al., 2003; Strack et al., 2008). But I think that in several aspects the proposed work will represent a step forward compared to these previous studies.
First, the proposed simulations will include a more advanced representation of the land surface, accounting for both biogeophysical and biogeochemical processes within vegetation and soil.
Second, the originality of the proposal also comes from the input land cover datasets that will be used in the simulations. This dataset is a newly developed representation of land cover dynamics in the EUS from pre-colonial times to present-day (Steyaert & Knox, 2008). Besides mapping land cover types, this dataset also provides information on biophysical properties associated with each land cover type (albedo, leaf area index, canopy height, rooting depth, surface roughness, etc.). This will enable the study of the sensitivity of climate to change in individual surface parameters, which is necessary to understand the physical mechanisms behind the overall effect of land cover change.
Third, previous mesoscale model simulations of the impact of land cover change on EUS climate have only focused on summer climate. This represents a very limited assessment of land cover change impact. In this project, the PIs propose to examine the impact of land cover change not exclusively on summer climate, but on the full seasonal cycle. From my point of view that represents the main strength of the project.
The proposed experimental design is ambitious, consisting of 20 different 1-year experiments (combining different land cover maps, background CO2, normal, wet or dry years and change in individual parameters). My only frustration about this experimental design is that each experiment has only one single realization (Perhaps would it be reasonable to run a smaller set of experiments but using an ensemble approach?). Previous experiments have shown that the climate signal arising from land cover change is often of small magnitude and sometimes even of opposite signs across different models. Careful statistical analysis is thus required when one wants to extract the signal due to land cover change from the noise. In that respect an ensemble approach would have been very useful.
However, I note that the PIs propose a way to address uncertainties in their simulations using the Maximum Likelihood Ensemble Filter (MLEF). I acknowledge that I have no expertise in this technique, therefore I can hardly tell if it is a relevant choice for the above- mentioned purpose.
What are the broader impacts of the proposed activity?
This project will contribute to the training of a PhD student. It should also benefit the scientific community by disseminating the results through conferences and publications. Furthermore, the PIs propose to make their model results available to the scientific community, which will probably stimulate further analysis by other groups.
By addressing aspects that have not been investigated in previous studies, I believe that this project will advance our understanding of land cover change impact on regional climate.
Ultimately, understanding how past land cover change did affect regional climate should provide guidance for future decisions concerning land management.
Summary Statement
This proposal builds upon strong existing bases and proposes to investigate several original aspects. The outcome from this research can be expected to broaden our understanding of land cover change impacts. That will be valuable for the scientific community and also ultimately for society, because this would help in the decision- making process concerning future land management.
Review #4
Proposal Number: 0840826
Performing Organization: U of Colorado Boulder
NSF Program: Climate and Large-scale Dynamics
Principal Investigator: Lu, Lixin
Proposal Title: Sensitivity of Weather and Climate in the Eastern United States to Historical Land-Cover Changes Since European Settlement
Rating: Fair
REVIEW:
What is the intellectual merit of the proposed activity?
The PIs both have a good background in the proposed research. They are highly productive scientists (as evidenced by their publication
records) and have made important contributions to the science. However, I have several concerns about the proposed research.
I do not see evidence of any new, creative, or original concepts. The research is simply an exploratory expedition using a particular model. The PIs plan 30 1-year simulations with various changes in the land surface or atmospheric CO2 concentration to see what happens. They plan to change land cover for various points in time (1992, 1920, 1850, 1650), change CO2 concentration, and then individually change specific land surface parameters (e.g., albedo, roughness, LAI, rooting depth). There are no new ideas that will greatly alter our understanding of land-atmosphere interactions.
The robustness of these results is not clear, or even considered in the proposed work plan. How much will the results depend on specific model formulations? The sensitivity to soil moisture is very important. This is discussed in little detail, except to say that simulations will be performed for normal, wet, and dry years. There is no discussion of ensemble simulations. Little thought is given to model initialization and how this initialization will effect the simulations (1-year in length). Soil moisture will be initialized from offline simulations of SiB2 driven by reanalysis data, but how sensitive will the 1-year simulations be to this?
The budget is very large . a total of $657,000 over three years . for the amount of results that will be gained. It supports 7 months of the PIs’ salary, 2 months of a professional research assistant, and 1 Ph.D. student.
What are the broader impacts of the proposed activity?
The proposed research will train 1 PhD student.
Summary Statement
The PIs have a strong research record, but this proposal has been put together in a very cursory manner. New, creative, or original concepts are lacking.
—————————
Most Recent Reviews
Date: Tue, 2 Feb 2010 12:35:44 -0500
From: “Torgersen, Thomas”
To: lixin@cira.colostate.edu, pielkesr@cires.colorado.edu
Subject: Collaborative Research: Sensitivity of weather and climate in the
Eastern United States to
Drs. Lu and Pielke,
I attach the reviews and comments on your recent proposal. At this point, I would admit that the budget is tight and this is falling on the “not enough funds” side of things. However, I have one more idea that I will pursue however much a stretch it is…
Nonetheless, I wanted you to have the reviews and comments available if you were planning on submitting a (category 3) WSC proposal.
PROGRAM OFFICER COMMENTS
0943712 Lu and Pielke
Collaborative Research: Sensitivity of weather and climate in the Eastern United States to historical land-cover changes since European Settlement
$534,372 Lixin Lu Colorado State University
176,532 Roger A Pielke, SR University of Colorado at Boulder
The goal of this project is to determine the sensitivity of weather and climate to historical land-cover changes in the eastern United States since the arrival of European settlers. Regional Atmospheric Modeling System
(RAMS) coupled with the Simple Biosphere (SiB) model will be used to perform a series of one-year ensemble simulations over the eastern United States with the present-day and several past land-cover distributions. The land-cover distributions will be based on the new Reconstructed Historical Land Cover and Biophysical Parameter Dataset developed by Steyaert and Knox
(2008) which uses 36 land-cover categories to describe the landscape for 4 time slices, 1650, 1850, 1920, and 1992 in the United States east of 97ºW. This project will rework Strack et al 2008 (land cover caused 0.3-0.4C temp
increase) with a better model. The observed land-cover and land-use changes and the climate system feedbacks associated with these changes, are expected to have important ramifications for the eastern U.S. weather and climate, as well as energy, water, and carbon budgets. Understanding the associated consequences requires a modeling system capable of accounting for the relatively high-resolution land cover and vegetation biophysical parameter changes, and the related changes in energy, moisture, momentum, and carbon fluxes which is within the capability of RAMS/SiB and is capable of simulating critical atmosphere, ocean, and land-surface interactions
AD HOC REVIEWER COMMENTS (direct quotes)
Reviewer 1 GOOD
…the methods, model, and scenario design seem well-suited to addressing the questions… I would not rate the proposal as transformative (most proposals are not), but it does represent a solid next step in the research this group has been doing…. This is a solid scientific proposal on a very relevant topic, a logical next step in the research this group and others have been doing using an interesting new data set on historical land use, and a very competent research group. The budget seems a bit high, the broader impacts are routine at best.
Reviewer 2 VERY GOOD
I do have some concern with whether the proposal is transformative. I do not believe that idea or tools proposed are transformative because the proposal is simply to run a regional weather model for different land use parameterizations. However, the results from this work do have the potential to be transformative if, as the authors hypothesize, the changes in temperature and precipitation due to land use change are found to be comparable to those expected from the IPCC multi-decadal global model simulations… The primary weakness that I see is related to the broader impacts
Reviewer 3 R
it is not clear if the nutrient effects on LAI are included in the analysis…
Reviewer 4 VERY GOOD
The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. … The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined… The authors list several impacts, some of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are providing a baseline to determine how future land-use chance can alter climate, and to ‘increase our understanding of the mechanisms by which land-cover change affects the atmosphere’
PANEL COMMENTS (direct quotes)
The stated goals are to understand the mechanisms by which land cover affects climate systems and to articulate the need for better land-cover representation in climate models. While the later is well known, the former has not been well articulated in the proposal… there is no effort to integrate the nutrient cycle. This was seen as weakness. Another weakness identified was the use of representative years for simulations to make climatological inferences. No effort is made to articulate how “representative” are the representative years… most parameters are fairly simple to obtain.
PROGRAM OFFICER ANALYSIS:
The project represents the next logical step in the analysis of land-use and climate variability through time over the eastern US. The hope would be that the model would “define” the role of land-use change and the residual might be indicative of climate change impacts. While this comparison can be made and will yield some results, the concern is that the results/contribution would be incremental; this remains high risk because it cannot be determined IF the two impacts can be separated. If the residual is small, does that mean climate change is small impact or that the model has wide error for land-use change? Given that this project merges existing models and datasets, it would be more beneficial to implement this effort on the path towards something more substantial e.g. parameter sensitivity assessment of RAMS/SiB and/or sensitivity to patch size of land-use or runoff variability as a function of land use and climate change and wet vs. dry. In its present design, it certainly does represent the next logical step but at a cost that it has not adequately justified in terms of the product to be produced. The results could be high payoff but at the present formulation, the risk is also high. The reviews and the panel analyses are generally weaker than fundable. I would urge the PIs to stretch themselves further with each proposal.
Thomas Torgersen 2 February 2010
Program Officer, Hydrologic Sciences
FULL REVIEWS AND PANEL SUMMARIES FOLLOW
Reviewer 1
Rating: Good
Review:
What is the intellectual merit of the proposed activity?
This research aims to use a new research product (historical land cover for eastern US in 1650, 1850, 1920, 1990, including biophysical parameter reconstruction (from Steyaert and Knox 2008); in the SiB-RAMS model to look at the impact of land cover change on annual and seasonal weather and climate. They pose an interesting question û were these changes (magnitude yet to be determined) of the same size (or larger) than projected 21st Century climate change? In 2008 Strack, Pielke, Steyaert and Knox published what must be a preliminary study for this proposed work: ‘sensitivity of June near surface temperatures and precipitation in the Eastern United States to historical land cover change since European settlement’ (it would have been good to see an exciting result from this study). This proposed project will expend this to look at variation through the year (seasonality) and also carbon balances in addition to weather variables.
The objectives are clearly stated and the methods, model, and scenario design seem well-suited to addressing the questions.
The group is highly qualified to do the research; they have extensive experience with the SiB-RAMS model (and other RAMS and RAMS/other models), have a solid (or better) record of publishing their results. Pielke has been a leader in this field for decades.
I would not rate the proposal as transformative (most proposals are not), but it does represent a solid next step in the research this group has been doing.
It is not clear exactly how much work there is and if it justifies 6 months per year of PI Lu’s time, a full-time graduate student, plus 1 month of Pielke’s time (reasonable) and 2 months for a Pielke assistant. There is very little model development proposed; the 30 simulation studies seem fairly straight-forward, though computationally demanding, to implement. These will require some attention and time to make sure the model keeps running and the results are saved, etc. etc. There will be a large amount of model output to process and analyze, presumably primarily by the graduate student, with supervision and assistance from Lu and Pielke.
The computing requirements for this project seem large – one 365-day simulation run is 24 wall-clock CPU days (I’m not exactly sure what this means?), so 30 scenarios = 720 wall-clock CPU days = 2 yrs (if nothing goes wrong). What if computing time not awarded by NCAR (maybe this is routine for these researchers)? Do they have sufficient computing available in that case?
What are the broader impacts of the proposed activity?
The broader impacts of this study are
(1) Scientific results on a topic (impacts of land use on climate) that is of interest and relevance to society and to a larger community scientific community. These results will be presented at scientific meetings and published in the scientific literature, and available on-line, and can be expected to contribute relevant and high-qualify science to the field of climate and global change.
(2) Education and training of a PhD student, in an important, interdisciplinary scientific field. The proposal also states that ‘other students and researchers, including those from underrepresented groups, will be trained in the use of the model that is applied in our study,’ but it is not explained how this will occur.
Summary Statement
This is a solid scientific proposal on a very relevant topic, a logical next step in the research this group and others have been doing using an interesting new data set on historical land use, and a very competent research group. The budget seems a bit high, the broader impacts are routine at best.
Reviewer 2
Rating: Very Good
Review:
What is the intellectual merit of the proposed activity?
This is a very well conceived, written, and organized proposal. The project team seems to be qualified to carry out the proposed work given their past experience in atmospheric modeling. Understanding impacts of land use change on weather and climate is an important science question, and the land use change dataset available to the researchers will allow for original work on quantifying the impacts of land use change from 1650 to present day.
I do not have experience with the models mentioned in the proposal and cannot speak to their applicability to the stated research hypothesis. To my mind, the authors have demonstrated that the models are appropriate for addressing the research objectives outlined in the proposal, although I would defer to an expert in climate modeling on this point.
I do have some concern with whether the proposal is transformative. I do not believe that idea or tools proposed are transformative because the proposal is simply to run a regional weather model for different land use parameterizations. However, the results from this work do have the potential to be transformative if, as the authors hypothesize, the changes in temperature and precipitation due to land use change are found to be comparable to those expected from the IPCC multi-decadal global model simulations.
What are the broader impacts of the proposed activity?
The proposed work will fund a PhD student who will be exposed to interdisciplinary studies including the use of cutting-edge models and observations at one of the nation’s best earth system science programs. The results will be disseminated through journal publications, conference presentations, and the project team’s own webpage. A potential impact to the larger society is improved understanding of how regional land use changes can alter climate systems.
Very little is mentioned about broadening the participation of underrepresented groups, other than mentioning that underrepresented groups will be included within the educational goals (with no specifics on how this will be done). The proposal does not address how the work will enhance the infrastructure for research and education.
Summary Statement
This is a well formulated and presented proposal by a seemingly qualified team that has the potential to offer important scientific understanding relating to the impact of land use change on climate systems. The primary weakness that I see is related to the broader impacts and the lack of a clear plan for involving underrepresented groups, for promoting teaching, training, and learning, and for enhancing research and teaching infrastructure.
Reviewer 3
Rating: R
Review:
What is the intellectual merit of the proposed activity?
This is a project to examine the effects of historical land-cover changes in the eastern US on land-atmosphere interactions and regional climate. They will assess their results relative to those from existing climate models from the IPCC, compare the magnitudes of changing land-cover effects among seasons, and evaluate the relative contributions of various biophysical factors (albedo, LAI, roughness, rooting depth, soil water saturation) on their predicted changes in climate and weather.
The project is ambitious and important; examining the effects of past land use change on local climate is a prerequisite for predicting and managing the effects of future land use change. There seems to be a lot of work proposed, but this is the team with the expertise to do it and their management plan indicates a well thought out approach and schedule.
My only concern is the carbon-water focus of the study. In a region so thoroughly worked over by agriculture, especially the initialization and then abandonment of agriculture, the effects of nitrogen and phosphorus are likely to be very important. Some of these effects might be accounted for through LAI, but it is not clear if the nutrient effects on LAI are included in the analysis. In any case, there are other effects of nutrients on plant and soil function other than changes in LAI, e.g., fertilization can enhance microbial respiration, increase plant production per unit leaf area, change rooting density and depth distribution, etc.
What are the broader impacts of the proposed activity?
Broader impacts:
It is clear that this type of work is vital if society is going to understand the impacts of climate change and adapt to it. The requisite mention of students and underrepresented groups is mentioned but not elaborated on (although I think one of the PI’s qualifies in the latter category). There is a request for support for 3years support for a graduate student(s).
Summary Statement:
See above.
Reviewer 4
Rating: Very Good
Review:
What is the intellectual merit of the proposed activity?
The proposal describes a fairly typical set of simulations meant to determine the effect of anthropogenic changes to the land surface on the North American climate. By including runs with different concentrations of CO2, the ultimate goal is to determine how land surface changes and anthropogenic changes in greenhouse gas concentrations interact to contribute to the observed changes in climate. The authors also propose to alter specific features of the land surface (root depth, albedo, etc.), so that the causes of climate change due to land surface changes can be determined.
Although the proposed research isn’t exactly unique (the authors cite a lengthy list of previous experiments, at least one of which (Baidya-Roy et al., 2003) also used RAMS to study the effect of historical land use changes on the North American climate), the current proposal seeks to build on earlier work by using newer surface data and a newer land surface model, as well as by running a more exhaustive set of simulations to explore many aspects of land surface change on climate.
Pros
The authors have identified a gap in the current understanding of past climate change and its implications for the future. They are quite clear about what they hope to accomplish, what experiments they wish to run, and how they will be carried out (for example, all will use modern reanalysis data to force the mesoscale model, and all will use a 10-15 year spin-up period). Both authors have experience with this type of modeling, and the proposed work can be done using existing, available models, as well as an existing land surface dataset. The timeline is reasonable and seems to allow enough time for all the proposed simulations, particularly if each run takes about 24 days to run on a single node. Given the list of well-planned experiments, it is clear that they will have a good idea as to what the relative magnitudes of the effects of land surface parameter changes will be.
Cons
In the original version of this proposal, the authors were vague about their use of ensemble modeling. Now, they have said more about it, mentioning that they will run each experiment 10 times. They are also more definite about their use of MLEF to determine the model uncertainty based on the ensemble results. The only problems I foresee are that they will get the uncertainty in the remote sensing products ‘through a literature review’, which assumes that the needed data is indeed published and available. Have they checked this? Also, the MLEF process is rather involved, and I think a more detailed description of processes like the Monte Carlo analysis at the flux tower is called for.
What are the broader impacts of the proposed activity?
The authors list several impacts, some of which are fairly standard (training a Ph.D. student, publishing the results, etc.), but, in my opinion, the most important are providing a baseline to determine how future land-use chance can alter climate, and to ‘increase our understanding of the mechanisms by which land-cover change affects the atmosphere’. These are important to the issue of how anthropogenic changes in greenhouse gas concentrations have altered climate in the past and how they will do so in the future. The 2007 IPCC Synthesis Report (IPCC, 2007) indicates that the effect of past land use change has been relatively modest, but also that the level of understanding is ‘medium-to-low’, so there is room for improvement in the understanding of these processes.
IPCC, 2007: Climate Change 2007: Synthesis Report. Contribution of Working Groups I, II and III to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change [Core Writing Team, Pachauri, R.K and Reisinger, A. (eds.)]. IPCC, Geneva, Switzerland, 104 pp.
Summary Statement
In deciding on the rating, I gave greater weight to the intellectual merit, since this is where the proposal’s greatest strengths lie. The proposal for the most part well-written, and spells out a clear, detailed research project. I was very intrigued by the research and would also be very interested in reading the resultant paper.
Panel Summary
In this proposal the PIs wish to take advantage of a new biophysical parameter dataset produced by Steyaert and Knox to perform coupled land-atmosphere model simulations to examine how the coupling between land and atmosphere has changed over a period of time. In particular they will examine the effect on temperature and precipitation in the Eastern US. The use of the dataset for the stated goal was seen as compelling. The strategy is to perform simulation over climatologically representative years. The stated goals are to understand the mechanisms by which land cover affects climate systems and to articulate the need for better land-cover representation in climate models. While the later is well known, the former has not been well articulated in the proposal. Although the subject matter is certainly timely and results could be highly significant if input data are sound, there does not seem to be a significant technique development component.
The reviews are generally supportive of the proposal. However, over such a long period of time, 1600s to 2000, the nutrient cycle undergoes significant changes, particularly in response to the land cover change. While the carbon cycle is incorporated in the description, there is no effort to integrate the nutrient cycle. This was seen as weakness. Another weakness identified was the use of representative years for simulations to make climatological inferences. No effort is made to articulate how “representative” are the representative years in the context of the interannual variability prevailing for the time period of study and the results can be used to make broad generalizations. If Steyaert and Knox maps and parameters are dependable, the proposed research could provide significant insight into land-atmosphere interactions and carbon cycling. However, most parameters are fairly simple to obtain. Does simply running the model justify the budget? There isn’t much detail on how under represented groups will be served.
Climate Science: Roger Pielke Sr. 