Archive for the ‘NIH Reviewers’ Comments’ Tag

Top Ten Things NIH Reviewers Should NOT Say In A Review

ID-10046929

Credit: Ambro at FreeDigitalPhotos.net

The Center for Scientific Review publishes their Peer Review Notes three times a year, and the most recent issue came out yesterday. The news items are always interesting and it is worth subscribing, if you don’t already. This issue contained an item about things NIH reviewers should not say. I repeat the list in its entirety here—I thought it might be fun for my grantees to see reviewers critiqued for a change.

What do you think of this list? Have you seen one or two of these on your Summary Statements? Me personally? I have seen variations on # 2, 4, and 10 in Summary Statements, and have strongly suspected reviewers of #1 and 5. I almost fell out of my chair laughing when I read # 7, sometimes I think CSR is a little out of touch with what actually happens on Study Sections:

  1. “I didn’t read the application, but I scanned it and saw the applicant said XXX. He doesn’t know what he’s doing.” Damning statements like this can skew a review discussion over something that might be insignificant in the context of the overall application. It’s better for you to ask other reviewers who have read the application carefully what they think about XXX.
  2. “This New Investigator does not appear to be fully independent since he continues to co-publish with his fellowship mentor/department chair, or does not have designated lab space, or has not been promoted in the past several years.”  Academic research organizations have widely diverse policies for faculty advancements and lab space, and many PIs maintain productive and healthy collaborations with mentors for many years after establishing themselves as bona fide investigators. You should focus more on the investigator accomplishments, such as being the first or senior author on a significant publication or giving presentations at major scientific meetings.
  3. “This application is not in my area of expertise . . . “  If you’re assigned an application you feel uncomfortable reviewing, you should tell your Scientific Review Officer as soon as possible before the meeting.
  4. “I don’t see this basic science research affecting my clinical practice any time soon.” An application does not necessarily have to show the potential for clinical or timely impact—if the applicant doesn’t make such claims. Basic research often takes time to pay off, and you’re charged to assess the “likelihood for the project to exert a sustained, powerful influence on the research field(s) involved.” Absence of an effect on public health does not necessarily constitute a weakness in basic science.
  5. “I like this project but I’m giving it a poorer score because the applicant has too much money.” Other funding is not a scoreable matter. You should focus on the application’s scientific and technical merit. However, you can note an excessive budget request in the budget section for NIH to consider.
  6. “This application has 2 great aims and 1 bad one. I would recommend deleting Aim 3, and I can give it a 1 or 2.” You cannot trade aims with scores. The application needs to be evaluated as a whole.
  7. “This R21 application does not have pilot data, which should be provided to ensure the success of the project.” R21s are exploratory projects to collect pilot data. Preliminary data are not required, although they can be evaluated if provided.
  8. “The human subject protection section does not spell out the specifics, but they already got the IRB approval, and therefore, it is ok.” IRB approval is not required at this stage, and it should not be considered to replace evaluation of the protection plans.
  9. “This application was scored a 25 and 14th percentile last time it was reviewed . . . .” You should not mention the previous score an application got, because this could skew the review discussion. Focus on the strengths and weaknesses of the current application as well as the responses to previous critiques.
  10. “This is a fishing expedition.” It would be better if you said the research plan is exploratory in nature, which may be a great thing to do if there are compelling reasons to explore a specific area. Well-designed exploratory or discovery research can provide a wealth of knowledge.
Advertisements

Reviewers of NIH Grant Submissions May Pay A Heavy Price When Their Own Submissions Are Reviewed

Grant scores are being posted right now. I was at a policy meeting last week with one of my grant clients and we waited with baited breath to see her scores. Although she has had a great deal of success on past NIH submissions, she was worried about this one, and with good reason. She recently agreed to serve as a reviewer on an NIH study section, but quickly realized that it meant that the resubmission of her own R01 competing renewal could no longer be reviewed in the most appropriate study section. Instead, it was assigned to a Special Emphasis Panel, or SEP. My client studied the list of reviewers on the SEP and learned that there was no one on the panel with the expertise needed to review her submission. She informed her SRO of this problem, but was told to sit tight and wait and see how the review went.

My client is considered an eminence in her field. She chairs a nationally-ranked academic department in her area, has landed numerous R01s, a U01, ARRA funding, and more. On her R01 competing renewal she scored in the 20th percentile, but the problems identified by the reviewers were certainly fixable and she resubmitted with hope, if not confidence. Then she accepted the position on the study section and the resubmission went to the (underqualified) SEP. The results of the resubmission were posted: Unscored.

It is hardly the first time I have heard such a story, it is just the most recent. It is considered an honor to be invited to serve on a study section. Many, like my client, choose to do so despite the time and effort involved because they feel they should give back for the many years of funding they have received from NIH. But the inadvertent result may be that the reviewer’s own submissions must be reviewed elsewhere, often on a panel that lacks the necessary expertise. As a result, my client and others have decided to terminate their service on study sections, which has the result of denying other NIH grantees reviews from those deemed the most qualified to provide them.

The current policy creates a situation where reviewers of a study section may be unable to get a fair review of their own grant applications at the most appropriate study section, thereby essentially penalizing them for service to NIH.

When discussing this problem last week in Washington with an NIH program officer, I was told to contact my local congressman. The PO felt that if we wanted to effect change to this NIH policy, the only approach was to inform a congressman that his constituents were not receiving millions in federal funding due to this policy. I plan to do so, and urge you all to do the same.

Posted October 30, 2012 by Meg Bouvier in Biomedical research, medical grant writing

Tagged with , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

The Value of the NIH Program Officer

I am starting to come up for air after having survived the first half of the Cycle II NIH grant deadlines—We have submitted all our new proposals (with the exception of an R34, an R15, and a lingering R01 with a rolling deadline because one of the PIs is an NIH reviewer.) Next come all the resubmission deadlines. Why do so many people submit Cycle II grants, as compared to Cycle I and Cycle III? Every person I know who handles NIH grants is exhausted, sleep-deprived, and crabby. Our families are ready to disown us. I am fantasizing about my vacation time on Cape Cod this August.

Now that half the deadlines have passed I will be returning to my usual blogging schedule. So what can I report from “the trenches” of federal grantwriting that will help prospective grantees? I will be in Washington at the end of this week to spend time with old friends from NIH, one of whom is a Program Officer. It got me thinking about how important POs are in my line of work. One recurring theme in my discussions with clients has to do with the value of the Program Officer to one’s grantsmanship. A good PO is worth their weight in gold. They are your conduit to the NIH. They can provide critical information about the institute’s funding priorities and how your idea may or may not fit into them. They are the only person involved in the review of your proposal with whom you are allowed any contact (remember, the second level review occurs at Council, where I have seen POs passionately go to the mats on behalf of a grantee in whom they believe.) Possibly the single most important thing you can do to improve your grantsmanship is to develop a relationship with your PO. I encourage grantees to draft a one-page Specific Aims, and email it to potential POs at various institutes that might have relevance to your work. Ask them if your project interests them and fits the funding priorities of the Institute, and whether you are considering the correct funding mechanism. Try to get them to discuss your project idea over the phone, and take copious notes when you do. If you listen carefully to what they say, you will garner a wealth of information. POs can provide a level of input that can literally save a grantee a resubmission, if the grantee listens carefully and heeds the POs suggestions. Remember, getting funded is not simply a matter of doing great science, but doing the sort of science in which the agency is interested. The PO can help you figure out how to alter your project to fit their priorities.

Take opportunities to contact them with questions as you write the proposal. For example, you might want them to provide clarification on some aspect of the funding opportunity announcement, or perhaps you want to ask permission to be considered an early stage investigator because the early years of your research were not in an environment that allowed proposal writing, such as industry. It always pays to call and ask such questions. There are few hard-and-fast rules at NIH, and such rules tend to differ (sometimes wildly) from Institute to Institute.

After submission, your relationship with the PO remains important. After review, call them to discuss your pink sheets and how to approach a resubmission, if they recommend one. If funded, keep in close contact with them throughout the year about progress such as publications or patents, and if problems arise with the project call the PO for advice immediately. (Don’t wait for the annual progress report to share bad news, it just pisses them off.) If your project is going extremely well, you are hitting your milestones and cranking out publications, ask the PO if there are any internal funds to extend the project a little longer as you put together a renewal application. The worst they can say is no.

If you develop an established relationship with the PO, invite them to lunch or coffee the next time you are in Washington—but don’t offer to pay, as that is not allowed. I am not suggesting icky lobbying-type behavior on your part. Your goal here is information exchange. You each have important information to share with the other. I have a very personable, likable client who is passionate about his work. He has a very well established and close relationship with his PO. When his R01 scored in the 30th percentile, the PO dipped into her own internal pot of money (via an R-series grant mechanism that is not open to external applicants) and gave him two years of funding to help beef up his preliminary data before he resubmitted his R01. That’s how powerful an advocate your PO can be.

Remember that speaking with prospective grantees is a large part of the job description of an NIH PO. Most POs are extremely helpful in this regard. In addition, they tend to keep their jobs for a long, long time (would you give up the chance to help set national biomedical science funding priorities, regularly discuss cutting-edge research with scientists in the field, and enjoy federal benefits?) I have clients whose relationship with their PO spans years, even decades. They can be an invaluable critic and advocate for your work.

 

 

The Importance of the “Approach” Criterion On NIH Grant Scores

Sally Rockey, Deputy Director of Extramural Research at NIH, posted data on her blog concerning reviewer behavior on proposals reviewed in 2010. Recall that with the new NIH grant format, NIH implemented scores of one to five (lower is better) in each of five categories: Significance, Innovation, Approach, Investigators, and Environment, as well as the overall impact score. Of 54,727 applications reviewed in FY10, 32,546 were discussed and received overall impact scores. The NIH generated some data on these scores. Among their findings:

*While reviewers used the full scoring range (one through five) for each of the five review categories, their scores were distributed more widely for the Approach category.

*Criterion scores are moderately correlated with each other and with the overall impact score.

*Of the five scoring categories, the one best able to predict the overall impact score was the Approach section (followed by significance, innovation, investigators, environment.)

The language surrounding the changes to the NIH grant format in Zerhouni’s efforts to enhance peer review stressed the importance of Impact, Significance, and Innovation. And in the new format, the length of the Research Strategies was halved, forcing grantees to compress “Approach” sections such as the exhaustive literature review and the detailed methodologies. However, from the data in Rockey’s blog post, we might surmise that reviewers still heavily weigh the Approach category. And based on my own experience with pink sheets in the new format, reviewers’ nearly-insatiable desire for preliminary data appears to continue unabated, despite the reduced page limits on proposals.

Multiple Regression To Predict Impact Scores Using Criterion Scores

Criterion Regression Weight
Approach

6.7

Significance

3.3

Innovation

1.4

Investigator

1.3

Environment

-0.1

%d bloggers like this: