In the spring of 2010, the National Science Foundation (NSF) created a task force on the merit review process in order to align the review criteria with NSF’s new strategic plan. The final recommendations from that task force included two review criteria and five review elements, all of which were outlined during a presentation by Miami University professor of biology and former NSF program officer Joyce Fernandes on November 11. Fernandes’ talk,, “NSF resubmission: How to decipher the panel summary,” was part of OARS’ fall NSF workshop series.
Fernandes says that when evaluating proposals, all reviewers are asked to evaluate against two criteria:
- Intellectual merit (potential to advance knowledge)
- Broader impacts (potential to benefit society and contribute to the achievement of specific, desired societal outcomes)
Five key elements are considered when evaluating both of the review criteria. These elements include:
- The potential for the proposed activity to advance knowledge and understanding within the field (or across different fields) and benefit to society.
- The extent that proposed activities explores creative, original, or potentially transformative concepts.
- A well-reasoned, well-organized plan for carrying out proposed activities.
- Qualifications of the individual, team or institution to conduct proposed activities.
- Adequate resources to carry out proposed activities.
“During a typical NSF review process,” Fernandes explains, “reviewers are selected based on their knowledge of the specific content area. Each proposal is typically assigned to three reviewers, with one acting as the primary reviewer.”
Fernandes goes on to explain that after discussing strengths and weaknesses of the proposal in regard to the Intellectual merit and broader Impacts, the panel makes a recommendation to place the proposal in one of 3-4 categories that indicate priority for funding (high, medium, low), or how competitive the proposal was (outstanding, highly meritorious, meritorious, non-competitive) . The panel summary, which is provided to the Principal Investigator (PI) along with the reviews, is a record of the discussion of a proposal and is intended to provide a rationale for the category in which the proposal was placed. PIs usually receive a context statement that tells how many proposals were submitted and reviewed, along with the ranking system (categories), and percentage of proposals that were placed into the categories.
“Bottom line: funding of proposals is based on dollars available,” Fernandes says. While program officers try to balance their portfolios based on things like emerging areas, broadening representation/participation, unique approaches to research questions, and transformational advances in the field, the number of proposals that are recommended for funding is based on the availability of program funds.
“At times, a lower ranked proposal might get funded over a more competitive proposal,” Fernandes says, because of an unusual merit of the proposal. She says it’s important for PIs to remember this as they review their summary statements.
“Almost all proposals submitted are meritorious, so it doesn’t take much to get your proposal kicked out of competition,” Fernandes says.
It is important that investigators who are responding to panel feedback take it seriously. Most investigators (after they have had time to simmer) find the panel feedback to be valuable as they rework their next submission.
Suggestions Fernandes offers for responding to the panel summary include reading the summary, taking time away before responding, re-reading and digesting the summary, discussing next-steps with the program officer, and if appropriate, resubmitting the proposal. (It should be noted that technically there is no category for “resubmissions.” All proposals to the NSF are considered new submissions).
Fernandes advises PIs who are still unclear about the feedback after reading and re-reading the panel summary to contact their program officer for clarification.
Finding the right program and understanding and responding to reviewer feedback are essential for increasing your chances for a successful submission. However, it is important to note that panel compositions change every year, and a new group of proposals is submitted for each deadline; therefore PIs should not expect that simply responding to reviewer feedback will move their proposal into a higher category.
Written by Tricia Callahan, Director of Proposal Development, Office for the Advancement of Research & Scholarship, Miami University.