The National Science Board has sent out a notice requesting input on the NSF review process. Their request is reproduced in this post. One glaring issue that is missing is accountability. I discussed this subject in my posts
I have made the following recommendations:
- Guarantee that the review process be completed within 6 months [my most recent land use and climate proposal was not even sent out for review until 10 months after its receipt!)
- Retain all e-mail communications indefinitely (NSF staff can routinely delete e-mails, such that there is no record to check their accountability)
- Require external independent assessments, by a subset of scientists who are outside of the NSF, of the reviews and manager decisions, including names of referees. This review should be on all accepted and rejected proposals ( as documented in the NSF letter at the end of this post, since they were so late sending out for review, they simply relied on referees of an earlier (rejected) proposal; this is laziness at best).
The National Science Board request follows. I will be submitting my comments, based on the above text, and urge colleagues who read my weblog to do likewise.
National Science Board
June 14, 2011
Over the past year, the National Science Board (NSB) has been conducting a review of the National Science Foundation’s merit review criteria (Intellectual Merit and Broader Impacts). At the Board’s May 2011 meeting, the NSB Task Force on Merit Review proposed a revision of the two merit review criteria, clarifying their intent and how they are to be used in the review process. In addition, the Task Force identified a set of important underlying principles upon which the merit review criteria should be based. We now seek your input on the proposed revision and principles.
The Task Force looked at several sources of data for information about how the criteria are being interpreted and used by the NSF community, including an analysis of over 190 reports from Committees of Visitors. The Task Force also reached out to a wide range of stakeholders, both inside and outside of NSF, to understand their perspectives on the current criteria. Members of NSF’s senior leadership and representatives of a small set of diverse institutions were interviewed; surveys about the criteria were administered to NSF’s program officers, division directors, and advisory committee members and to a sample of 8,000 of NSF’s Principal Investigators (PIs) and reviewers; and the NSF community at large was invited to provide comments and suggestions for improvements through the NSF web site ( http://www.nsf.gov/nsb/publications/2011/01_19_mrtf.jsp). The stakeholder responses were very robust—all told, the Task Force considered input from over 5,100 individuals.
One of the most striking observations that emerged from the data analyses was the consistency of the results, regardless of the perspective. All of the stakeholder groups identified similar issues, and often offered similar suggestions for improvements. It became clear that the two review criteria of Intellectual Merit and Broader Impacts are in fact the right criteria for evaluating NSF proposals, but that revisions are needed to clarify the intent of the criteria, and to highlight the connection to NSF’s core principles.
The two draft revised criteria, and the principles upon which they are based, are below. Comments are being collected through July 14—we invite you to send comments to firstname.lastname@example.org. It is expected that NSF will develop specific guidance for PIs, reviewers, and NSF staff on the use of these criteria after the drafts are finalized. Your comments will help inform development of that guidance, and other supporting documents such as FAQs.
The Foundation is the primary Federal agency supporting research at the frontiers of knowledge, across all fields of science and engineering (S&E) and at all levels of S&E education. Its mission, vision and goals are designed to maintain and strengthen the vitality of the U.S. science and engineering enterprise and to ensure that Americans benefit fully from the products of the science, engineering and education activities that NSF supports. The merit review process is at the heart of NSF’s mission, and the merit review criteria form the critical base for that process.
We do hope that you will share your thoughts with us. Thank you for your participation.
Ray M. Bowen
Chairman, National Science Board
Director, National Science Foundation
Merit Review Principles and Criteria
The identification and description of the merit review criteria are firmly grounded in the following principles:
- All NSF projects should be of the highest intellectual merit with the potential to advance the frontiers of knowledge.
- Collectively, NSF projects should help to advance a broad set of important national goals, including:
- Increased economic competitiveness of the United States.
- Development of a globally competitive STEM workforce.
- Increased participation of women, persons with disabilities, and underrepresented minorities in STEM.
- Increased partnerships between academia and industry.
- Improved pre-K–12 STEM education and teacher development.
- Improved undergraduate STEM education.
- Increased public scientific literacy and public engagement with science and technology.
- Increased national security.
- Enhanced infrastructure for research and education, including facilities, instrumentation, networks and partnerships.
- Broader impacts may be achieved through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by the project but ancillary to the research. All are valuable approaches for advancing important national goals.
- Ongoing application of these criteria should be subject to appropriate assessment developed using reasonable metrics over a period of time.
Intellectual merit of the proposed activity
The goal of this review criterion is to assess the degree to which the proposed activities will advance the frontiers of knowledge. Elements to consider in the review are:
- What role does the proposed activity play in advancing knowledge and understanding within its own field or across different fields?
- To what extent does the proposed activity suggest and explore creative, original, or potentially transformative concepts?
- How well conceived and organized is the proposed activity?
- How well qualified is the individual or team to conduct the proposed research?
- Is there sufficient access to resources?
Broader impacts of the proposed activity
The purpose of this review criterion is to ensure the consideration of how the proposed project advances a national goal(s). Elements to consider in the review are:
- Which national goal (or goals) is (or are) addressed in this proposal? Has the PI presented a compelling description of how the project or the PI will advance that goal(s)?
- Is there a well-reasoned plan for the proposed activities, including, if appropriate, department-level or institutional engagement?
- Is the rationale for choosing the approach well-justified? Have any innovations been incorporated?
- How well qualified is the individual, team, or institution to carry out the proposed broader impacts activities?
- Are there adequate resources available to the PI or institution to carry out the proposed activities?