The reviewing process from the NSF perspective.
This is taken from the NSF DEB Blog (slightly abbreviated):
For a while now we’ve been pondering how to approach a delicate subject: writing reviews. This subject is something of a minefield of tensions, conflicts, opinions, and opportunities to offend, alienate, and otherwise ruffle feathers by implying, “you’re doing it wrong.” So we feel it is important to explain here why we are tackling this topic and mention some of the approaches that didn’t fly.
We receive requests from less experienced reviewers who want advice outside the trial-by-fire of an actual panel to hone their review-writing skills. And we also hear from PIs who are disappointed in the utility or quality of the reviews they have received.To set the tone, let’s make a couple of blanket statements straight off. You are all volunteers in this effort and you do a great job and deserve immense gratitude. With that said, probably every established PI who has submitted NSF proposals has received at least one review that was… a bit less than they expected; perhaps lacking a certain degree of usefulness; or even a complete enigma.
What is going on with these sub-par reviews? Well, it’s difficult to address because we don’t think there is a singular problem. Simply put, in managing the whirlwind of research life, we may occasionally fall short in creating the ideal, critical, insightful and helpful proposal review.
Why are we addressing this here? There is quite a bit out there about writing manuscript reviews, and even some nice posts about PIs coming to grips with the reviews received. But there’s a pretty big void when it comes to discussion of actually writing proposal reviews. As a program office, we have a unique platform to start a discussion around review expectations and feel it is a topic worth talking about openly. What we hope to achieve is not a static prescription for a “good review” but for this to start a discussion, raising community awareness about the importance of proposal reviews and the need for continual improvement in writing them, regardless of how well we think we’re doing already.
There are many paths to a good proposal review. But the major commonalities can be distilled into a few points.
Good proposal reviews:
Of course the real challenge is, “How do I do it?” To help out we’ve started a list (in step-wise order[i]) of tips to guide you past the most common pitfalls on the path to completing a great review.
Notes: EVALUATIVE means using terms that express an opinion about the subject like “good”, “bad”, “excellent”, “inadequate”, “exemplary”, “satisfactory”, etc. SUPPORTED means following the evaluative term with an explanation like “because the [specific information in the proposal] is/is not reflective of [some important external knowledge like cited literature, best practice, or other relevant information].”
This is taken from the NSF DEB Blog (slightly abbreviated):
For a while now we’ve been pondering how to approach a delicate subject: writing reviews. This subject is something of a minefield of tensions, conflicts, opinions, and opportunities to offend, alienate, and otherwise ruffle feathers by implying, “you’re doing it wrong.” So we feel it is important to explain here why we are tackling this topic and mention some of the approaches that didn’t fly.
We receive requests from less experienced reviewers who want advice outside the trial-by-fire of an actual panel to hone their review-writing skills. And we also hear from PIs who are disappointed in the utility or quality of the reviews they have received.To set the tone, let’s make a couple of blanket statements straight off. You are all volunteers in this effort and you do a great job and deserve immense gratitude. With that said, probably every established PI who has submitted NSF proposals has received at least one review that was… a bit less than they expected; perhaps lacking a certain degree of usefulness; or even a complete enigma.
What is going on with these sub-par reviews? Well, it’s difficult to address because we don’t think there is a singular problem. Simply put, in managing the whirlwind of research life, we may occasionally fall short in creating the ideal, critical, insightful and helpful proposal review.
Why are we addressing this here? There is quite a bit out there about writing manuscript reviews, and even some nice posts about PIs coming to grips with the reviews received. But there’s a pretty big void when it comes to discussion of actually writing proposal reviews. As a program office, we have a unique platform to start a discussion around review expectations and feel it is a topic worth talking about openly. What we hope to achieve is not a static prescription for a “good review” but for this to start a discussion, raising community awareness about the importance of proposal reviews and the need for continual improvement in writing them, regardless of how well we think we’re doing already.
There are many paths to a good proposal review. But the major commonalities can be distilled into a few points.
Good proposal reviews:
• are conceived with an open mind and reflect serious consideration of the proposal
• make a compelling case for the evaluation, citing evidence or examples as needed
• are well-written and direct
• contain no ad hominem criticisms or unsupported assertions
• Refresh your expectations, EVERY TIME: You are asked to judge a proposal against a specific set of review criteria. These review criteria are not static. They change over time; some change almost yearly and may have changed since you last looked. They vary by organization; each successive layer of directorates, divisions, and programs has an opportunity to modify and specify review criteria; what you are asked to do in education or geosciences will differ in the specifics from what you are asked to do in DEB even though all still abide the NSF merit review criteria[ii]. They can be stacked and combined; even within a single panel some proposals will include additional review criteria relevant to a submission option or solicitation on top of those of the conceptual program at the heart of the proposal.
• Recognize your audiences (yes, plural): You, as a reviewer are always writing for two, and potentially three, audiences. Firstly, the review is to inform us (NSF) as to why you think the proposal is or is not worth funding; we consider this in light of others’ reviews and our award portfolio to arrive at a funding recommendation. Secondly, the review is to help the PI learn how to prepare a better proposal next time, or to improve the project if it is funded. Lastly, the written review may be used as documentary evidence that you as the reviewer gave the proposal thorough and thoughtful consideration.
• Don’t summarize, review: Every audience for the review will also have access to the actual proposal and doesn’t need you to summarize it; we need your well-argued opinion of it. Note: Starting out with largely descriptive text may be important to organizing your thoughts as a reviewer. That’s ok. That’s great. But it’s not the end point. What we’re saying here is: take whatever part of the proposal you felt important enough to describe and fill it out by giving your opinion of it and telling us why you hold that opinion.
• Provide clear and substantiated comments: To satisfy the distinct audiences and avoid summarizing, every point you make in your review should exhibit these 3 characteristics:
• It is EVALUATIVE
• It is SUPPORTED with appropriate details, evidence, and comparisons
• It provides CONSTRUCTIVE feedback in the context of the proposal
• Be self-critical of your critical comments: A good critical comment delves below the surface of your initial reaction and constructively reflects the opportunities and constraints for addressing the issue. This step could easily be its own separate post; we’ll provide a longer discussion of this at a later date. For now though, the fast summary is: see the flaws, point them out but give them context, and don’t get hung up on small stuff. Stop and ask: why is it a flaw for this proposal? Do I fault others for this consistently? If not, why have I pointed it out now? Is there a deeper cause of this flaw that I can describe? (If so, do it.) Search out your biases. Recognize and differentiate between actual problems in the proposal and legitimate differences from how you would do the work. Make sure your suggestions for improvement are clear and reflect the constraints on space: it’s limited, and PIs can’t include “more” material on X without cutting Y. A useful suggestion needs to identify both the X that needs expansion and the Y that could stand to be cut.
• Minimize or omit purely descriptive text: This is a repeat of point 3. We’re serious about this. It is very easy to fall into the summarizing trap – it’s much easier to write that material – and we have to remind ourselves about this constantly too. If some aspect of the proposal is important enough for you to describe, tell us how it affects your overall opinion of the proposal and why.
No comments:
Post a Comment