Delusional Biomedical Researchers Seek Repeal Of Arithmetic

February 14, 2011

There is a letter being sent around via e-mail that expresses the views of biomedical researchers who are pushing a petition to repeal the new NIH rule that only a single resubmission of an unfunded competing grant application is permitted. Here is the text of the letter that I received via an e-mail forward:

Dear Colleagues,

I am writing to solicit your help in changing a new NIH policy that I
believe will have an enormous negative impact on our field. As most of
you know, a recently adopted rule states that if a grant proposal is not
funded on the first submission, only one revision can be submitted with
the same specific aims. If that revision is not funded, the proposal
must be “substantially” changed. As far as I understand, the rule was
adopted to discourage “serial resubmitters”. While such a policy could
make sense in an era of reasonable paylines, with the projected budgets
rumored to be funding at the 7th percentile in some sections, this could
have a dramatic and I would argue devastating effect on the research
efforts in this country. Consider the following:

The rule will have a disproportionately negative impact on young
investigators with early stage and therefore less diverse programs, or
more senior investigators who also have more narrowly focused programs.
How can a young investigator, for example, who is just starting
“substantially” change their aims when they have to focus their efforts
on a very limited number of projects undertaken with limited funds and
staff. These people are often hired by senior faculty on the basis of
their first projects and to be told they must change on the basis of
applications that might fail despite being ranked better than 90% of
grants submitted, seems patently absurd. And worse, it is likely to be
profoundly discouraging and destructive.

All of us who have sat on study section know that we cannot distinguish
a 15th percentile grant from a 5th percentile grant. It is simply beyond
the resolution of the process. Therefore, this new rule will have the
consequence of redirecting the science of many of our very best
scientists on the basis of what will essentially be an arbitrary
criterion.

The meaning of “substantially changed” has not been clearly defined.
Program Officers themselves are not sure what this term means and are
not being given adequate guidance. I have heard things from “51%
different”, change the tissue or cell type you are working on, any aim
included in either the first application or revision cannot be included,
etc. We need clear and unequivocal guidance on this point, and I would
argue we need it immediately as “new” applications are being prepared by
a large number of investigators at this time.

The alternative that I advocate would be to go back to a system where at
least 2 revisions of the same application would be allowed. While we
will still obviously lose some superb applications if the pay line stays
where it is, I think this would provide a much fairer assessment of the
research proposals received by the NIH.

My intention is to let the feelings of a large number of scientists on
this subject be known. If you are willing to sign an email that will be
sent to both Francis Collins and Tony Scarpa (Director of Center for
Scientific Review) that raises these points, please let me know by
simply responding to this email and (if possible) forwarding it to 10
people who you know (not on the current recipient list) that might also
want to sign. If I can accumulate a large enough number of signatures
(100-500, say) I will draft a letter and send it first to all who have
expressed interest in signing to get feedback.

I must say, I am not generally prone to such activism but I think things
have just gotten to the tipping point.

I look forward to your responses.

There is a lot of fundamental misunderstanding going on in this letter. We can start with this:

As far as I understand, the rule was adopted to discourage “serial resubmitters”.

No, the rule was adopted to try to stop the common study section behavior of putting meritorious applications in a “holding pattern” for one or two resubmissions, so they can fund the applications that have already been in a holding pattern. The other change made by many ICs to prevent the “holding pattern” is the adoption of different paylines for A0 and A1 applications, with the former substantially more generous.

More importantly, however, there is a serious delusion that underlies this letter. There is only so much money available to fund competing applications, and the only effect changes in peer review in terms of actual funding of such applications could possible have is a change in which applications get funded. So the notion of “meritorious applications going unfunded because of this pernicious new rule” is nonsense. Limiting resubmissions can’t possibly change the number of “meritorious” applications that go unfunded.

The letter authors seem to have forgotten that–while they may feel put upon that they only get a single resubmission–all their competition also only get a single resubmission. The playing field is still even, but in a context that should make peer review more efficient by substantially reducing “holding pattern” study section behavior. It will also reduce the PI behavior in response to “holding pattern” of submitting half-baked proposals they *know* aren’t fundable in order to “get in line” in the “holding pattern”.

Finally, as far as New Investigators/Early Stage Investigators being disproprortionately affected, that is a red herring. Their grant applications are reviewed separately in study section, and are supposed to be assigned criterion and impact scores by comparison with each other, not with established investigator grants. And even more saliently, all ICs have substantially more generous ESI paylines than for established PIs, in some cases more than *twice* the established PI percentile cutoff.

This new restriction on resubmissions can’t possibly lead to an increase in the number of “meritorious” applications that would have been funded under the previous system now going unfunded. The laws of arithmetic cannot be repealed, and these researchers would be much better advised to devote their energies to lobbying Congress to support the NIH budget, not tilting at irrelevant peer review windmills.

UPDATE: Since there already seems to be substantial confusion about what I mean by “the laws of arithmetic cannot be repealed”, the point is that the number of grants that can be funded is zero-sum. Putting more applications in a holding pattern can’t possibly do anything about that.

The purpose of the one-resubmission rule and the more generous A0 payline is to try to identify the most most meritorious grant applications as efficiently as possible and to get those grants funded as soon as possible. All the A2s that no longer exist and thus cannot be funded means that an exactly equal additional number of A0s and A1s *do* get funded.

About these ads

41 Responses to “Delusional Biomedical Researchers Seek Repeal Of Arithmetic”

  1. Todd Says:

    Sure you make good points, but you did not comment on the issue of young investigators having a disproportionately more difficult time having multiple lines of research so that they can have new specific aims for the next application with such limited resources. With paylines like this the only solution is to have multiple pans on the iron. Harder for ESI! That to me is the part of the letter that resonates.

  2. Pascale Says:

    In the study sections I have been on, new applicants are scored within the other proposals, not separately. We are asked to keep the new investigator status in mind, and the PO can pull some strings when it goes to Council.

    I have never seen the noobies reviewed separately.

  3. drdrA Says:

    No, the rule was adopted to try to stop the common study section behavior of putting meritorious applications in a “holding pattern” for one or two resubmissions, so they can fund the applications that have already been in a holding pattern.

    And what exactly was wrong with that behavior. What is wrong with holding over an application in the 9th percentile, versus scrapping it and starting all over with a ‘new’ application?

    ‘and these researchers would be much better advised to devote their energies to lobbying Congress to support the NIH budget’

    See here- I agree with you. But do you know a SINGLE scientist that has lobbied Congress. I mean, A SINGLE ONE? Seriously. Let’s look at ourselves on this one. Yeah, that includes you.

    Actually I think this is less about lobbying congress right now than about the piss poor job that scientists have done at bringing to the public why tax $$ should be spent on research. Congress people are supposedly only doing what their constituents elected them to do- and the current crop were elected to unfund basically everything that isn’t military.


  4. I’m not sure when was the last time you have served, but as part of the Enhancing Peer Review initiative (and coincident with the move to the 1-9 scoring system?), study section order of review has been mandated to be (1) in descending order of preliminary impact score and (2) with NI applications reviewed in a so-orded group separate from established PI applications.

  5. BugDoc Says:

    It is indeed deluded to think that investigators who don’t get funded after 2 tries in this economy can continue for one application after another to come up with “new” applications. Especially since no one including the NIH program officers seem to be entirely clear on what “new” means. Even more so since reviewer burnout is resulting in less “expert” reviewers for each proposal. Is this measure sufficient? Of course not. Lobby Congress? Absolutely. Bring grassroots common sense to NIH? I hope so…eventually. Guess we won’t count you in though – you’ll be too busy spreading delusional and irrelevant platitudes like “the laws of arithmetic cannot be repealed”. Clever.

  6. drugmonkey Says:

    And what exactly was wrong with that behavior. What is wrong with holding over an application in the 9th percentile, versus scrapping it and starting all over with a ‘new’ application?

    c’mon drdrA, this is not hard. The problem was that study sections don’t bother to look at the A0, waste of everyone’s time. The only versions that get taken seriously are the A1, A2. This is supposed to be a “fish or cut bait” instruction to the study sections. All else you can chalk up to the budget.

    It is indeed deluded to think that investigators who don’t get funded after 2 tries in this economy can continue for one application after another to come up with “new” applications. Especially since no one including the NIH program officers seem to be entirely clear on what “new” means.

    And the A2 makes all the difference in the world on this? c’mon…..

  7. drdrA Says:

    DM- I suppose there is some Jeremy Berg data that I’m missing about how no one looks at A0s? Please cite for those of us who clearly missed it.

  8. drugmonkey Says:

    But do you know a SINGLE scientist that has lobbied Congress. I mean, A SINGLE ONE?

    Both of you do.

  9. drugmonkey Says:

    The funding data on A0 vs A1 vs A2 from the doubling through the GreatNIHDepression are pretty damn clear evidence in support of the more anecdotal reports from study section and PPs description of a holding pattern. I’ve posted graphs on more than one occasion.
    http://scienceblogs.com/drugmonkey/2011/01/initial_outcome_of_limiting_ni.php

    The only alternative is that the pool of researchers suddenly lost their natural minds and couldn’t write grant apps without the help of the first round of review….

  10. drdrA Says:

    Thanks for the link.

    Ok, so I know a single one then. I’d say that is pretty freaking rare behavior among academic scientists.

  11. BugDoc Says:

    Nobody thinks bringing back the A2 will fix everything. And as stated in the letter, at reasonable funding levels, allowing only 1 resubmission would probably be a positive thing. The problem is that at these funding levels, NIH coming up with more restrictions on peer review, more “special” funding opportunities, more big translational science grants and centers and constant tweaks to the application and review format are just exacerbating an already very stressed system. The grassroots effort by Benezra represents one step forward. It may be a long path, but you have to start somewhere. The scientific community can’t fix the budget, but we can at least participate in reforming the system.

  12. drugmonkey Says:

    an exactly equal additional number of A0s and A1s *do* get funded.

    Are you listening people? *your* A0 or A1.

    Now, the funny thing in all of this is that the original letter writer is a highly established PI. There’s no way in hell that he is going to be unable to create new Aims, probably with lots of data too. And while I have no idea about him personally, this concern about the ESI seems a bit like crocodile tears. (yeah, I have a post brewing)

    What really *will* start taking it in the chin is the competitive renewal. All those old folks who reify the continual refunding of their project and swing dangly bits at each other over the -25 or -30 appended to their project are the ones who are scared witless. Over this completely irrelevant point-scoring marker, I might add. I think the biggest dent is going to be made in the population of endlessly renewed projects….

  13. drugmonkey Says:

    you have to start somewhere.

    Yes but you should start somewhere that isn’t idiotic.

    NIH coming up with more restrictions on peer review,

    what? how so?

    more “special” funding opportunities,

    Allowing investigator-initiated to go to A2 doesn’t do a damn thing to pull money back from RFAs…

    more big translational science grants and centers

    …or to stop this trend.

    constant tweaks to the application and review format are just exacerbating an already very stressed system.

    It may be annoying to have come through the shortening and changed review criteria but 1) the major lifting is over and 2) what does it really change in terms of your personal success?

    Look, everybody is scared of the same damn thing. Less money means steeper odds for them. and everybody identifies “problems” and designs solutions that work to their own personal advantage. I’m actually pleasantly surprised that this letter writer mouthed so many good things about ESI but…where the hell was he (and all of his fanboi/grls) when things were going so poorly for newbs while established investigators were making out like bandits? hmmm?


  14. [...] see more on this debate, check out BlueLabCoats for the pro-side and CPP for the con side.  The scientific blogosphere awaits an a response from the venerated Drugmonkey. [...]

  15. BugDoc Says:

    Actually, I’m in favor of shortened applications and the new review criteria. At least they got something right. I also think it’s appropriate that ESIs get a bump in payline, since the shortened applications may work against newbies. There is no single solution that’s going to fix the big pile of shit we’re all squatting in. Of course, bringing back the A2 doesn’t fix money going to Centers, etc, but it’s disingenuous and self-defeating to imply that if one solution doesn’t fix everything, then it’s worthless.

  16. Spiny Norman Says:

    “Actually I think this is less about lobbying congress right now than about the piss poor job that scientists have done at bringing to the public why tax $$ should be spent on research.”

    Absolute horseshit. Public opinion polls show that the American public overwhelmingly supports S&T research, and especially medical research. The general public does not need convincing.

    The current problems are more systemic, and reflect (1) low ability to do long-term planning because of budget volatility; (2) a neanderthal fuckstick contingent that just took over the House in the midterm elections, which actively despises knowledge generally, scientific knowledge in particular, and hates scientists with a special passion. These assholes did not win because of their views on S&T research, which were orthogonal to the midterm election.

    So the general public does not need convincing, and the GOP/Tea Party is unconvincible. If you want to see better funding for S&T research, you’ll be wanting to work to eject the know-nothings. That is really our only hope.

  17. BugDoc Says:

    I’ve been trying to eject the entire Republican Party for my whole adult life. No luck yet though.

  18. arrzey Says:

    This reminds of me of the urban legend story to legislate the value of pi to be 3.

    Changing the number of resubmissions will NOT change the number of grants funded. It is unlikely to change WHO gets funded (do you really think No0bs are better at revision than old farts?).

    And… I find the slightly insidious remarks about DM & PP’s funding status on the amusing side. Read what they write, that speaks more highly to their commitment to changing the system than their current receipts from it.

  19. neurowoman Says:

    I understand why people are scared, but bringing back the A2 isn’t going to help the overall funding rates, which is what’s going to hurt people. For my part, I’d gladly trade getting funded a year earlier for having to write the same damn grant for the third time!

    As to the issue of what constitutes a “new” application, well, maybe a chat with your PO is in order. I don’t quite buy the complaint that you have to change your entire research line, since the smart PI’s I know have more than enough interesting ideas to last a lifetime, even within one research area. And most newbies get knocked for putting too much in their first grant (“overambitious”), so just put those in a separate grant! Big deal. Besides, if you don’t have some new ideas to work on after the 2+ years it takes to submit the A0 and then the A1, what the heck have you been doing?

    ESI’s will be helped by the no-A2 policy. I’ve known several PIs on the cusp of tenure before their A2 submission got funded. Getting funded a year earlier will ease the tenure stress greatly.

    I think we should let it run and if it’s not working, then consider change, but don’t bail before it’s even had a chance.

  20. neurowoman Says:

    And nobody should comment on this without considering the graph Drug Monkey posted above showing the trends in funding A0/A1/A2 applications…


  21. [...] on to my ramblings, you may wish to read a bit of pro/con debate from DrDra of BlueLabCoats and Comrade PhysioProf as a warmup. Additional from Prof-like Substance and Genomic [...]


  22. [...] by drdrA I’m delighted that the to A2 or not to A2 bomb in my last post is generating so much discussion all over the blogosphere. It is obvious that people have strong opinions about this [...]


  23. For ESIs/n00b PIs, eliminating the holding pattern is a good thing as you can get funding earlier rather than wasting time collecting more preliminary data for resubmissions when you could be working on the project.

    /from one who heard through the grapevine that her non-NIH grant was not funded because greybeard reviewers felt that n00bs should have to resubmit at least once.


  24. Granted that both systems are fair, and fund the same number of grants. Still:

    * Especially valuable proposals could lose their 3rd chance, when reviewers didn’t “get it”;

    * Applicants could further improve the same proposal after the 2nd rejection; and

    * The labor saved from getting rid of the 3rd try must be balanced against the cost to applicants who need to re-orient their work.

    Whatever their importance, the arithmetic argument doesn’t cover these issues.

  25. Anonymous Says:

    Yes, especially valuable proposals did loose their given 3rd chance because reviewers did not get it or did not want to get it.

    Any proposal can be further improve not only after the 2nd but even after the 6th rejection.

    The labor saved from getting rid of the 3rd try must be evaluated in the context of national biomedical priorities.

    Number of resubmissions will not change the arithmetics of funding but could change the quality of opportunities for a diverse biomedical workforce and bring rational change in how taxpayers moneys are used by grantees institutions. That is to be seen if the opportunity to test it is allowed.

  26. drdrA Says:

    Um. Spiny Norman-

    Obviously we disagree. The GOP/Tea Party folks got elected by someone. The people who elected the current GOP/Tea Party crop may say they are for research, but their actions say that they would rather not pay for it (or for anything that matter that isn’t military). I stand by my claim that we as scientists have done a poor job communicating why what we do is important, and why taxpayers should pay for it.


  27. Everyone’s proposal is especially valuable, and maybe its more of a case of the proposal sucking than the reviewer not getting it? The dynamics of having a research proposal sort of mandate flexibility. I’m sure many folks are still not looking for the same rabbis down the same exact hole from when they started. You redirect your research to where it is taking you, re-orienting is difficult, impossible no.

  28. skeptifem Says:

    Same old shit everywhere- instead of punishing the people who are fucking it up (in this case, serial submitters) they make the system extremely frustrating for everyone else via bullshit rules. I can’t think of a place I have worked that did not pull this gradeschool “I’ll just punish the whole class then” shit on adults. jesus.

  29. drugmonkey Says:

    Is there any evidence for “serial submitters”? meaning, one presumes, people who submit apps that have no chance whatsoever of funding?

    My experience in this regard is that the worst of the worst submit once and disappear. The “serial submitters” to a specific study section typically were decent researchers that eventually ended up with funding.

  30. Anonymous Says:

    It’ll be interesting to know how the term “serial resubmitters” was coined. The investigator initiating the “go back to A2″ must have picked it from somebody at NIH. It is not an attractive term because it sounds very much like “serial killers”. On the other hand, there is some/a lot of true behind the term. What about those investigators submitting competing renewal P0s for ever ?. I mean 20 years of a P0 with the same “state of the art”, “close-knit” and all that impressive jargon for the same stuff proposed 20 years ago can be certainly considered a more of the same ” serial resubmitter”….

    I mean call it what you want. In the end lots of money for “little new” under the sun. And, most likely, quite a number of casualties and losses.

  31. C Crowley Says:

    It’ll be interesting to know how the term “serial resubmitters” was coined. The investigator initiating the “go back to A2″ must have picked it from somebody at NIH. It is not an attractive term because it sounds very much like “serial killers”. On the other hand, there is some/a lot of true behind the term. What about those investigators submitting competing renewal P0s for ever ?. I mean 20 years of a P0 with the same “state of the art”, and all that impressive jargon for the same stuff proposed 20 years ago can be certainly considered a more of the same ” serial resubmitter”….

    I mean call it what you want. In the end lots of money for “little new” under the sun. And, most likely, quite a number of casualties and losses.

  32. Eli Rabett Says:

    You have two choices, birth control or fratricide. Either hold down the number of new doctoral students and postdocs or kill off the megalabs.

  33. JackDanielsBlack Says:

    There is an article in this week’s JAMA wondering why academics no longer assert leadership in clinical trials and are now “minor players carrying out someone else’s research, chairing steering committees or putting their names on manuscripts in which they have had little input”. (JAMA, Vol 305 No. 7, pp. 713-714). Maybe NIH should just go out of business and leave it all up to the pharmaceutical companies — looks like the grants they do give are often ineffectual.

  34. Andy Says:

    Where is the economics of clinical trials.gov?. Is it supposed to be at NIH Reporter ?. I just looked for some of the “completed” or “active but not recruiting” or “recruiting” and the info is scanty or nonexistent.
    Am I looking at the right place?

  35. Spiny Norman Says:

    And I reiterate: most people did not vote on this issue, and will never vote on this issue. The technical term for your plan is “futile cycles.”


  36. [...] NIH officials Sally Rockey and Larry Tabak have weighed in on the delusional hullabaloo surrounding the wise decision to do away with so-called A2s, second grant resubmissions, and to only allow a single resubmission of a particular grant application. For a recap of what we are talking about, see this previous post. [...]


  37. [...] point that Rockey doesn't make terribly well is one made by PhysioProf: More importantly, however, there is a serious delusion that underlies this letter. There is only [...]

  38. Frederick Sachs Says:

    The limit of one resubmission is not about funding more grants but supposedly to reduce the study section’s time. As those of us who have served on review panels know, it takes little time for a previous reviewer to rereview a resubmission. The net time saved the panel is not significant.

    However, if you assess the time required by an applicant to prepare a new grant with the requirement that for undefined reasons it has to be different from the last, the sot is enormous.

    A typical grant application requires at least four months, unless from a big lab where a postdoc is given the job. I one figures 4 months of the time of a trained faculty and the net pay rate is perhaps about $100k/year with an (NIH) overhead of about 50% the cost per grant is about $50k apiece, far more than the cost for a reviewer to reevaluate a revised proposal.

    Thus what we have do is to divert trained american personnel to do a job for much more money than it could be done abroad. Does this sound familiar? Reducing the United State’s ability to compete world wide? All of this from a supposed time saving of an hour or two by some reviewer. Way to go NIH!


  39. […] paying attention to such matters in 1997 but there was some screaming in 2009, let me tell you. Delusional Biomedical Researchers Seek Repeal Of Arithmetic More on the new NIH policy on grant application revisions Initial outcome of limiting NIH apps to […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 37 other followers

%d bloggers like this: