Hand tapping one star in a five-star review system.

Editor offers advice for handling a difficult peer review

We’re pleased to reblog this Duke University Press post by guest blogger Courtney Berger. Berger is an executive editor with a university press, so this post focuses on peer review of books under consideration for publication. However, most of her advice applies just as well to peer review of grant applications (just substitute “editor” for “program officer!”).


On a not-too-infrequent basis I see posts and memes in my social media feed denouncing the dastardly deeds of Reviewer #2—that querulous and impossible-to-please peer reviewer. I usually hover over the post, thinking that I might chime in with a bit of helpful advice. I am a book editor after all. Surely I can say something to help alleviate my friend’s experience of feeling misread, misunderstood, or even personally attacked by an anonymous peer reviewer/colleague. But I always resist weighing in, knowing that at that moment my friend just needs to voice their frustration and receive some affirmation. It can be painful to receive this kind of criticism, especially when facing the pressures of tenure and promotion. However, while momentarily painful, even a negative peer review can be a good thing, and you can use the report to strengthen your book. So, here’s a bit of practical and philosophical advice to help you work through a tough peer review.

1) Go ahead and vent—but be careful about where and how you do so.

As I mentioned, I see plenty of social media posts railing against Reviewer #2. No judgment. It’s good to get your community to support you through tough times. But I would caution against offering too much detail in a (semi)public forum or lingering in this phase for too long. It’s a small world—and although there should be an appropriate amount of distance between you and the reviewer, it’s always possible that they are in or adjacent to your social circles. You never know when the person you’ve declared to be the enemy of your book project will turn out to be the person you most wanted feedback from. (Yes, that happens!) After your initial venting, share the report with a trusted friend or colleague and get their feedback. Perhaps they will have a different take on the reader’s comments. They may identify productive advice that it was tough for you to see at first. If it helps, write a scathing response, voicing all of your frustration with the reader’s misapprehensions and misreadings. Get it all out. Then file it away.

2) Focus on problems, not solutions.

My colleague Ken Wissoker touched on this in his blog post on the merits of peer review, and it’s a strategy that I frequently employ to help authors shift their perspective on a review (even a positive one!). It’s easy to get hung up on the reader’s suggestions for how to improve your book. Maybe they recommend adding a chapter or including analysis of a topic or critic that you think is tangential to your project. Or, perhaps you feel like they didn’t “get” your argument or missed a point that’s already in the manuscript. Your job is to figure why the reader is tripping up. If you said something and they missed it, that may not be the reviewer’s fault. Chances are the point is buried at the end of a chapter or not articulated with enough force. In that case, you need to clarify and highlight your claims so that the reader does get it. It’s not uncommon to have two readers—one more positive, the other more critical—pointing to the same issue. It’s just easier to hear the person who presents their comments more constructively. As the author, it’s your job to make the leap and to figure out what your readers need in order to be convinced. Once you do that, it will be much easier to come up with a revision plan.

3) Clarify your vision.

Use the reader’s comments to sharpen your own vision for the book. I often ask authors early in the process: what do you want your book to accomplish? Are you aiming to shift a scholarly conversation, revise an accepted history, offer a new theoretical tool? Do all of the parts of the book support that mission? Clarity on this point will help you to decide which advice to take on board and which to leave by the wayside. The goal of the review process is to help you write the book you want to write, but even better. Let me repeat that, since it’s easy to forget as you’re wading through frustration, self-doubt, or any of the other feelings that this process provokes. You should use the review process to help you realize your vision for the book and to help you say what you want to say in a way that will reach your readers. For a peer-reviewed book, you need to do that in a way that is convincing to other experts in your field; but the book is yours. (Note: I am setting aside exigencies such as tenure review, departmental pressures, and disciplinary policing, which can make this more complicated. But I always urge people to come back to their own ambitions for the project. The audiences and conversations you initiate or enter into with the book are the ones you’ll likely be engaging with for a while, and so they should be ones you care about.)

4) Talk to your editor.

Sometimes a negative review might mean that a press decides to turn down your project, and you may not have an opportunity to get substantial feedback from the editor. But other times, if the reports indicate that the project has great promise, an editor might be eager to work with you to see the book to publication. So process the report, get through the venting phase, and then set up a time to talk to your editor or send them an email with your preliminary thoughts and questions. As the editor, I have a different perspective. First, I know who the readers are, and while I keep their identities anonymous, I can also help an author think critically about the book’s audience and why a particular reviewer might be frustrated with the manuscript in its current state. For example, maybe you thought the book was for a history of science readership. Reviewer #2’s comments might help you to realize that this audience won’t be as receptive to your work. Is this who you are really writing for? If so, you may need to make some adjustments. If not, you may need to reframe the book for the readership you want. Also, I appreciate authors who can take a tough criticism and respond productively. I take it as a good sign when an author is willing to tackle Reviewer #2’s comments and use the feedback to make their book even better.

5) Remember that the review process is part of a larger scholarly conversation.

For many the review process simply feels like a set of hoops to jump through. And it can be that. But it’s also a chance to learn from your peers—just as you would when presenting a paper at a conference—and to respond. While there is the occasional mean-spirited reviewer, most readers are trying to be helpful. Try to receive the comments in the same spirit. Be grateful that someone took the time to read and think with you and take what you can from the conversation.

6) Make your response about you, not the reviewer.

Your editor may ask you to write a response to the reader reports, addressing the readers’ questions and laying out a revision plan. It’s tempting to use this as an opportunity to demonstrate all the ways that Reviewer #2 was wrong. (See #1 above: if you do this, keep it in your drafts folder.) Instead, focus on what you plan to do to improve the book. Now is the time for solutions! For example, if the reader didn’t think the book’s argument was cogent, offer a clear and concise overview of the book’s intervention. If the structure wasn’t working, explain how you will either adapt the structure or make the structure more visible so that the reader will understand it. And hold your ground when you need to. If you really don’t agree with a reviewer’s take on your project, say so and explain how you will make your vision for the project come to life.

6) Know when to cut your losses.

Sometimes a negative review is just a negative review. As difficult as it sounds, you may need to set it aside and move on—to a new press or to a new reviewer, depending on the situation. But hopefully with some of these strategies you can get the most out of the review process, and maybe someday you’ll even be thanking Reviewer #2 in your acknowledgments!


Source: What to Do About Reviewer #2: Advice for Handling a Difficult Peer Review

One-star review image by Gerd Altmann via Pixabay. Reviewer 2 image by KotaYamaguchi via imgflip.

 

A hand grasping a magnifying glass reaches out from a computer screen.

Should I resubmit?

Error message on computer screen reads, "Try again later. Something happened at our end." There is a close button.

We’re pleased to reblog this Strategic Grantsmanship post by Kelly Byram.


Seasoned grant seekers know that thick skin and abundant tenacity often separate the funded from the unfunded, and resubmission is just part of the process. Of course, getting comfortable with the concept of the sunk cost fallacy and being willing to walk away from untenable (read: unfundable) proposals is also a skill that will optimize your return on your valuable time.

The question I often hear from those new to grants is, “How do I know whether to resubmit or move on?” Less-experienced PIs tend to take the review of their initial submission as the final word on the matter and abandon an unfunded proposal, whereas seasoned PIs know that resubmissions generally have a much higher success rate than new submissions. At the NIH, the success rate for new (A0) R01s in 2015 was just 13.1%. That same year the success rate for (A1) resubmission applications was 33.5%.

Source: National Institutes of Health

So how do you know if you should resubmit? The 10 questions I ask to assess whether a proposal should be resubmitted are:

  • What score (or percentile) did the proposal receive? What was the payline? This provides a quick assessment of the proposal’s competitiveness.
  • Does the funding opportunity allow for resubmission? If not, does the proposal align with another funding opportunity? If the answer to both of these questions is “no,” then, unless something changes, your time is probably better spent pursuing other projects.
  • What are the team’s thoughts on the reviewers’ comments? If the proposal is a multiple-PI/PD proposal, the question of submission has multiple primary stakeholders, but feedback from the whole team (not just the PIs) is incredibly valuable in judging if a proposal is worthy of resubmission.
  • Was your application triaged or streamlined? Often funders will triage applications that clearly do not align with the funding opportunity. Many review processes then involve reviewers scoring applications and streamlining (i.e., removing from the process) the bottom 50% of applications from the process. If your proposal was triaged or streamlined for misalignment, you may want to consider if the proposal would fare better with a different funding opportunity with which it is in better alignment.
  • Was the reviewer response generally enthusiastic? Reviews of proposals can be harsh (even when the proposal earns a good score!), so much so that Reviewer #2 has become a meme on social media frequented by the academic set. I advise applicants to read the summary statement once, put it down for a while, then read it a second time with an eye to assessing the level of enthusiasm and important issues that impacted the score. Regardless of the technical issues which may or may not be addressable (see below), if the enthusiasm for the project isn’t there, you need to assess if that is because the project is not a good fit for the opportunity or if it has a likeability issue that can be addressed in a resubmission.
  • Are you willing to fully address reviewers’ concerns? Sometimes it is apparent that there has been a misunderstanding of a part of the proposal and a simple clarification will clear the matter right up. Other concerns may require a change to the design, personnel, or budget.
  • What would be the new timeline, and will it work for the research? If a resubmission were to be successful, when would it be funded? At that point, would the team be available? Will other elements of the research (space, support personnel, community collaborators, etc.) be available? Remember to consider the timeline of other, complementary project funding.
  • Have you spoken to the Program Officer (PO) about the proposal? In addition to providing advice about a possible resubmission to the same opportunity, the PO may know of other opportunities for which your proposal is better suited.
  • Has very similar research been funded in the interim? If so, your proposal may have become unfundable.
  • Are the other members of your team still enthusiastic about the proposal? The time between a proposal’s submission and receipt of the Summary Statement can be long, and sometimes a year or more may have passed since the team began work on the proposal. A frank discussion among the members about possible resubmission should include a discussion of commitment to the project moving forward.

Many times, what it all boils down to is this: Are you willing to fully address the concerns outlined in the Summary Statement? If there is reviewer enthusiasm for the idea and you are willing to fully address the concerns, then moving forward is feasible. If you are not willing to address reviewer concerns articulated in the Summary Statement, then there is little point in resubmission. Remember the Golden Rule.

Still undecided? NIH advice on resubmission can be found here.


Source: Should I Resubmit?

Magnifying glass image by mohamad_hassan via Pixabay. Try again image by Roger Green via Flickr. Both used under Creative Commons license.

 

Former NSF program officer demystifies review and resubmission processes

An illustrated figure of a featureless person holds a giant red pencil, checking off boxes on the piece of paper on which he stands.

In the spring of 2010, the National Science Foundation (NSF) created a task force on the merit review process in order to align the review criteria with NSF’s new strategic plan. The final recommendations from that task force included two review criteria and five review elements, all of which were outlined during a presentation by Miami University professor of biology and former NSF program officer Joyce Fernandes on November 11. Fernandes’ talk,, “NSF resubmission: How to decipher the panel summary,” was part of OARS’ fall NSF workshop series.

Fernandes says that when evaluating proposals, all reviewers are asked to evaluate against two criteria:

  • Intellectual merit (potential to advance knowledge)
  • Broader impacts (potential to benefit society and contribute to the achievement of specific, desired societal outcomes)

Five key elements are considered when evaluating both of the review criteria. These elements include:

  • The potential for the proposed activity to advance knowledge and understanding within the field (or across different fields) and benefit to society.
  • The extent that proposed activities explores creative, original, or potentially transformative concepts.
  • A well-reasoned, well-organized plan for carrying out proposed activities.
  • Qualifications of the individual, team or institution to conduct proposed activities.
  • Adequate resources to carry out proposed activities.

“During a typical NSF review process,” Fernandes explains, “reviewers are selected based on their knowledge of the specific content area. Each proposal is typically assigned to three reviewers, with one acting as the primary reviewer.”

Fernandes goes on to explain that after discussing strengths and weaknesses of the proposal in regard to the Intellectual merit and broader Impacts, the panel makes a recommendation to place the proposal in one of 3-4 categories that indicate priority for funding (high, medium, low), or how competitive the proposal was (outstanding, highly meritorious, meritorious, non-competitive) . The panel summary, which is provided to the Principal Investigator (PI) along with the reviews, is a record of the discussion of a proposal and is intended to provide a rationale for the category in which the proposal was placed. PIs usually receive a context statement that tells how many proposals were submitted and reviewed, along with the ranking system (categories), and percentage of proposals that were placed into the categories.

“Bottom line: funding of proposals is based on dollars available,” Fernandes says. While program officers try to balance their portfolios based on things like emerging areas, broadening representation/participation, unique approaches to research questions, and transformational advances in the field, the number of proposals that are recommended for funding is based on the availability of program funds.

“At times, a lower ranked proposal might get funded over a more competitive proposal,” Fernandes says, because of an unusual merit of the proposal. She says it’s important for PIs to remember this as they review their summary statements.

“Almost all proposals submitted are meritorious, so it doesn’t take much to get your proposal kicked out of competition,” Fernandes says.

It is important that investigators who are responding to panel feedback take it seriously. Most investigators (after they have had time to simmer) find the panel feedback to be valuable as they rework their next submission.

Suggestions Fernandes offers for responding to the panel summary include reading the summary, taking time away before responding, re-reading and digesting the summary, discussing next-steps with the program officer, and if appropriate, resubmitting the proposal. (It should be noted that technically there is no category for “resubmissions.” All proposals to the NSF are considered new submissions).

Fernandes advises PIs who are still unclear about the feedback after reading and re-reading the panel summary to contact their program officer for clarification.

Finding the right program and understanding and responding to reviewer feedback are essential for increasing your chances for a successful submission. However, it is important to note that panel compositions change every year, and a new group of proposals is submitted for each deadline; therefore PIs should not expect that simply responding to reviewer feedback will move their proposal into a higher category.

Finally, understanding the NSF review process is one key to writing a successful proposal. You can learn more about the NSF proposal process here or by volunteering to serve on a NSF review panel.

Written by Tricia Callahan, Director of Proposal Development, Office for the Advancement of Research & Scholarship, Miami University.

Illustrations by AJ Cann via Flickr, used under Creative Commons license.

Looking up at several stories of office windows from inside a building. There is a round, black circle-shaped sculpture suspended from the glass ceiling of the building.

OARS offers fall workshop series

Head-and-shoulders portrait of a bespectacled woman in a tan top with a red and pink zig-zag pattern.
Former NSF program director and professor of biology Joyce Fernandes is one of the presenters for OARS’ fall workshop series.

This fall, OARS will host a series of brown-bag workshops on navigating the NSF proposal process.  Workshops will be held select Tuesdays from noon to 1:00pm in Pearson 208.  You are welcome to attend any or all of the sessions.

September 16
Writing an effective NSF proposal: what’s your sales pitch?
Led by Joyce Fernandes, Department of Biology
RSVP here.

September 23
NSF Graduate Research Fellowship Program: how do I apply?
Led by Tricia Callahan, OARS
RSVP here.

October 14
NSF broader impacts: integrating your research with educational activities
Led by Joyce Fernandes, Department of Biology
RSVP here.

October 21
NSF data management: what is data management, why is it important, and how do I write a sound data management plan?
Led by Eric Johnson, University Libraries
RSVP here.

November 4
Funding opportunities for STEM education
Led by Joyce Fernandes, Department of Biology
RSVP here.

November 11
NSF resubmission: how to decipher the panel summary
Led by Joyce Fernandes, Department of Biology
RSVP here.

November 18
Communicating with the NSF program officer: how, why, do’s and don’ts
Led by Joyce Fernandes, Department of Biology
RSVP here.

Featured photo (left) by Luke Faraone via Flickr, used under Creative Commons license.  Photo of Joyce Fernandes (above) by Miami University Photo Services.