Pages

Good For You

When reviewing an NSF grant proposal, the choices for rating the overall proposal are:
  • Excellent : Outstanding proposal in all respects; deserves highest priority for support.
  • Very Good : High quality proposal in nearly all respects; should be supported if at all possible.
  • Good : A quality proposal worthy of support.
  • Fair : Proposal lacking in one or more critical aspects; key issues need to be addressed.
  • Poor : Proposal has serious deficiencies.
The overall rating includes evaluation of both Intellectual Merit and Broader Impacts components of the proposal, but for most Science proposals, the Intellectual Merit score is typically the one that drives the overall rating.

Obviously it is important to get lots of Excellents, but it's possible to get a grant with some Very Goods. In some cases, it is possible to get funding with mostly E/VG but also a Good (or worse) if the negative reviewer amply demonstrates a lack of objectivity, knowledge, or sanity.

As is well known by those who have submitted an NSF proposal, a rating of Good is Not Good. Reviewers who interpret Good as indicating "a quality proposal worthy of support" are typically those outside the NSF system who think this description actually means what it says. It does not. Good is Bad.

A reviewer who has serious concerns about a proposal or a PI can do a lot of damage with a rating of Good, but Fair and Poor are the real killers. When is a rating of Fair appropriate? When is a rating of Poor appropriate? (Let's assume that the reviewer is objective and is not downgrading a proposal for evil reasons.)

In ~ 20 years of reviewing NSF proposals, I have only given a few ratings of Poor. From what I've seen on panels and other NSF committees, this is not just because I am nice; a rating of Poor is rare in my field of science. Also, I am not particularly nice.

For a proposal to be Poor, it has to be so bad that there is no hope that the proposed research will be successful in any way. A dull but otherwise solid proposal (i.e., "We are going to get some data but we don't really know why") is not Poor. Such a proposal might be Fair or Good, depending on whether anything of use or interest could be salvaged from such a study. A Poor proposal is apocalyptic. A Poor proposal has no redeeming qualities.

I have given more ratings of Fair than Poor over the years. Fair (in my opinion) indicates a flawed proposal but not one that generates feelings of anger or disgust at the stupidity of the proposed research. A Fair proposal involves poorly defined ideas and/or inappropriate methods. It is possible, but unlikely, that the research would result in anything useful or interesting, but awarding the grant would not be a major travesty; it would be a semi-travesty.

What if a proposal is not well written or organized but the research is not as bad as might seem based on how the proposal is written? That can be a difficult situation to assess. Does the bad writing and organization indicate a problem with the research design or implementation? Or are these aspects technical flaws that mask the excellence (or very goodness) of the research?

In these cases, I make a decision based on the research, but I may mention in my review the aspects of the proposal writing/construction that are problematic if it seems that these might impact the research. A proposal is not a manuscript, however, so I don't do any pointless technical editing.

If the PI has experience with writing proposals, the only technical flaws that are important to note are those that might indicate a problem with the research. An inexperienced PI, however, might benefit from advice about proposal writing; ideally this advice will be constructive and not patronizing.

Most proposals that I review are Very Good. A few are Excellent. More are Good or Good/Very Good (although split ratings are kind of lame). Some are Fair. Rare ones are Poor.

These ratings are important for determining whether a proposal is further considered, but an essential aspect of the review is the comments, which should match the rating. I recently read some reviews of one of my proposals and there seemed to be no correspondence of the comments and the ratings. Some of the most negative ratings had very positive comments; it was odd.

Just as with reviewing manuscripts, though, I know that my opinion is just one part of the process. If I rate a proposal as Poor or Fair (the equivalent of recommending rejection of a manuscript), I am not making the final decision to reject the proposal (or manuscript). I am providing my opinion, and this will be considered along with other input from other reviewers and panel/program officers (or editors, for manuscripts).

If my opinion is honestly given and is based on a careful reading of the proposal (or manuscript), then I don't feel bad about giving a negative rating. I might feel sympathy for the PI (author), but I was asked for my opinion, and I gave it, explaining the reasoning for my negative opinion. It's up to others to evaluate my review along with the rest of the review information and make a decision.