The following guest post was written by Dr. Carl Batt. Batt is Liberty Hyde Bailey Professor in Cornell University’s Department of Food Science. In his 30-year career, he has served on a number of NIH study sections. Below, Batt shares his experience with ancillary criteria that may affect how NIH proposals are scored in the review process.
Sitting on a study section (review panel) at NIH helps researchers understand (as much as possible) the nuances of how panels work and what the collective wisdom of the panel might be. For those of you not familiar, the standard practice is to have three or more preliminary reviewers whose scores form the basis for the selection of proposals that will (or will not) be discussed at the review panel. There, the lead reviewer presents the case, followed by the secondary and tertiary reviewers. If the proposal is not scored at the review panel, there is no reporting of these preliminary scores.
The following comments address a number of ancillary criteria that reviewers are asked to comment on and may (or may not) impact the preliminary scoring. Should the proposal move forward to the panel, these criteria are sometimes not discussed until after the reviewers have offered their final scoring. Each and every member of the panel (aside from those who have a conflict of interest) provide a score, which could be impacted by the discussion about these criteria.
There is no doubt that the research plan is the most important element judged by the reviewers, beyond the importance of the topic itself. There is no saving a bad idea, especially if it is deemed not to be innovative. Over the past few years a number of additional criteria have been introduced as the NIH (and other agencies) attempt to address a number of issues, some of which are external, and others internal. The following are things to keep in mind. While they will not sink a great proposal, they could potentially tank a borderline proposal. They are not scored but reviewers are asked to note compliance and challenges. What that translates into is reviewer concerns that could — if they get noticed and bother the reviewer — creep into the preliminary scoring (for the initial reviewers) and the overall impact score if your proposal gets scored.
Ancillary criterion 1: Rigor
Significant attention is being directed toward the reproducibility of research and this has crept into all science, not just that which involves qualitative, seemingly-subject experimental methods. Rigor is defined by the inclusion of controls, replicates, and other matters that are typically thought of as just “good science,” but that you might not include in an explicit manner. The Center for Scientific Review (CSR) manager (and sometimes the academic panel manager) will often remind the reviewers to comment on “rigor” and for the most part the answer is “affirmative,” meaning either there is an explicit mention of “rigor” in the experimental plan or the reviewers simply didn’t notice it. On rare occasions, the reviewer might comment about the lack of rigor and, as with everything else related to review, you are not in the room to defend your proposal. It is important that you explicitly mention controls and replicates to ensure that the matter of rigor is covered. In the past it might have been mentioned as part of a reviewer’s commentary but now it is an explicit part of the review process.
Ancillary criterion 2: Gender of animal or human subjects
If you propose to use animals or human subjects be aware there is a specific question asked: “Has the applicant considered ‘sex’ as a potential variable in their animal studies?” At a minimum you should state the gender of the animal or human subjects you will be using. It’s appropriate to state that you will use both male and female subjects, but you should follow that statement with an additional statement that you will consider gender as a variable in your experimental plan.
Ancillary criterion 3: Milestones
Along with other agencies, the NIH has moved toward a more rigorous set of expected outcomes. Some funding opportunity announcements (FOAs) have explicit language requiring milestones, and it isn’t a bad idea to include them even in proposals for FOAs that don’t require them. The following is the language found in one NIH FOA.
Milestones. This sub-section is required for all applications. All applicants must describe here a set of discrete benchmarks that will allow unequivocal determination of the progress made towards the goals of the project. Milestones should be scientifically justified and well defined for each year of the project and be based on the proposed specific aims. Whenever feasible, milestones should provide quantitative benchmarks for comprehensively assessing the annual progress of the project. Milestones must not be simply a restatement of the specific aims. The specific aims describe the research goals of the project. Rather, the milestones should provide the means for assessing the progress made towards each aim and offer a timeline and a “pathway” for the testing of a discovery concept or development of a technology. The completion of these milestones will be used to judge the success of the proposed research on an individual-project basis.
Examples of Milestones:
- Verify that the designed composite nanoparticles are able to reproducibly release an activated component at tumor/cancer cell sites in vivo.
- Ascertain that a new targeted nanoparticle can specifically deliver a therapeutic agent to the tumor by demonstrating that agent concentration in tumor exceeds at least “x” times its blood concentration.
- Demonstrate the ability of a nanoparticle diagnostic construct to detect at least “x” specific proteins in blood (out of “y” specific proteins proposed) at a femtomolar level.
- Demonstrate the ability of the proposed nanotechnology to achieve 95% rate of capture for circulating tumor cells in blood.
Reviewers have various opinions about milestones and most don’t understand exactly what they mean. But some do, and the simple advice is to have milestones that include quantitative outcomes that can be measured and promised with some degree of temporal specificity. In other words instead of “I will build you a house,” think “I will complete your 3 bedroom, 2 bath, 2000 square foot home by April 1, 2019 for $100,000.”
Ancillary criterion 4: Resource availability
This request has to do with how you will make available some of the outputs of your research, including data, but also including physical resources such as strains, plasmids, or other things. The simplest answer is to offer everything to everyone, but the practical reality might be different. The goal is to make sure there are no doubts about the availability of resources. The section should consider what and when you might make things available and under what conditions. It is certainly acceptable to include the prerequisite that a material transfer agreement (MTA) be in place.
Ancillary criterion 5: Resource validation/concern for authenticity
This section addresses how you will know that the reagents (animals, cell lines, chemicals) you are working with are what you think they are. This stems from reports that cell lines for example are frequently not what people say (or believe) they are. “I will buy stuff from established vendors,” is an answer, but sometimes reviewers have objections to that statement and will make note of it (and sometimes that might impact their initial scoring). Even established chemical vendors (e.g., Sigma) are the subject of some derision. There is little to be done, as nobody can afford to send every chemical out for independent analysis. Certainly a statement about the certificate of identity might be worth noting. In the case of cell lines, having some markers that you state will be assayed (especially where there are specific mutations or knock-outs) is a way to address this section.
Written by Carl Batt, Liberty Hyde Bailey Professor in Cornell University’s Department of Food Science.
HeLa cell image by EnCor Biotechnology via Wikimedia Commons. Mouse photo by Rama via Wikimedia Commons. Both used under Creative Commons license.