Yesterday The Conversation published an article titled “Science funding should go to people, not projects” which has as its leading hypothesis that we should shift from a project funding model to a person funding model. The reason for this is that it is difficult to compare the relative merits of projects from different fields, and in times of extremely low success rates leads to the preferential funding of “safe” projects (often projects which are essentially complete already). Immediately following publication of this article, the twitterverse exploded, with several very interesting reply threads arising (unfortunately the discussion became a bit ad-hoc, and no # was devised at the time, however @Dr_Mel_Thomson has created a great Storify page).
Let me say straight out that I am against funding people in the absence of projects. My concern (similar to those raised by several others) is it unfairly biases against young Early and Mid-career Researchers (EMCRs) and women. It encourages funding to go to senior academics in large established labs. To counter this, many will argue that track record will be judged ‘relative to opportunity’, and this leads me to the topic of this post, can we shift away from the current qualitative measure of track record to a more quantitative metric that more accurately reflects the idea of ‘relative to opportunity’?
With the rebuttal period for the major funding rounds for the Australian National Health & Medical Research Council (NH&MRC) and Australian Research Council (ARC) just about to finish, I’m sure everyone has their own anecdotal story to tell about the evaluation of track record. From a personal perspective, I’ve had comments suggesting I had a “moderate” track record from one assessor, and an “outstanding” track record from another – on the same grant. Regardless of what my track record may or may not be, these comments indicate the problems associated with a qualitative assessment of track record.
Assessors for NH&MRC are told to ignore journal impact factors and other metrics such as H-index when assessing track record, and they are told to assess the candidate relative to opportunity. In the absence of guidelines, this becomes haphazard at best (through no fault of the assessors, having assessed several grants over the last three years I find it tough). While it is easy to pick the investigators/teams with the absolute top track records (6/7’s for NH&MRC and A’s for ARC), and the non-competitive track records, the next tier down becomes difficult (think 5’s for NH&MRC.and B/C’s for ARC).
Imagine logging on to RMS (for ARC) or RGMS (for NH&MRC) and being able to quickly calculate a researcher score (lets go with the current NH&MRC system of 1-7). If your score comes out as less than 4.5 you are ineligible to submit a grant as a lead investigator. Straight away, this would reduce the burden placed on external reviewers by triaging applicants who are deemed not-competitive. A score of 4.5+ allows you to submit your project proposal for review. During the rejoinder process, RGMS/RMS allows you to update your track record and recalculate your score, at this point a minimum of 5.5 is required to progress. This would allow those researchers who have that big, exciting paper published in May/June to significantly improve their score.
Following these two rounds of culling based on track record, the remaining grants could go to the grant review panels (NH&MRC) or College of Experts (ARC) for discussion. Track record scores could then be adjusted by the respective panels when deemed necessary.
Now the big question, what variables do we plug in to generate the magical score? Can we limit the potential for researchers to game the system? Can we make the quantitative score fair across a broad range of disciplines? Importantly, any quantitative value/metric MUST improve on the current system. This is where I make my disclaimer, these variables are simply a list of things I would like to see incorporated, it is by no means exhaustive and there will be plenty I have forgotten (feel free to point out things I’ve missed in the comments), but these are things I believe are important. Any score should be a combination of both track record (numerator) and time/opportunity (denominator).
NUMERATOR VARIABLES
FIRST/LAST AUTHOR PUBLICATIONS – First and foremost, any track record score needs to take in to account both the quantity and quality of publications. Generating a fair score for this is difficult, impact factors can bias against certain smaller fields, citations are not necessarily a true indication of the impact of a piece of work, Altmetrics can be gamed (to some extent). A significant improvement in recent times are the inclusion of article level metrics (how many times an article has been viewed, downloaded, cited). Each one of these variables is imperfect on its own, but a combination of journal impact factor, article level metrics, and online impact (altmetrics) could give a reasonable measure of the “impact” of the work.
SECONDARY AUTHOR PUBLICATIONS – The number of first/last author manuscripts are particularly important, but we also need to recognise the importance of secondary author manuscripts as well. Science is supposed to be collaborative, and the last thing we want to do is encourage researchers to work in isolation, and yet so many reviewers count up the number of first and last author manuscripts and ignore everything else. This removes the impetus for collaboration. Could the publication metric be weighed by position in the author list? (1.0 for first/last, 0.2 for secondary author?).
PREVIOUS FUNDING – As you’ll see, I’ve included this in the calculation of the NUMERATOR and the DENOMINATOR below, as funding often begets more funding. But it is important to recognise a track record of being able to successfully obtain funding, from national funding agencies, international sources and industry.
PRIZES/AWARDS – At the moment I’m not convinced this section really adds much to the track record score, as assessors tend to focus on publications above all else. But prizes and awards are important, they are often awarded by our peers in recognition of outstanding work. Again, difficult (but not impossible) to put in to a numerical score. Any scoring system should take in to account whether the prize/award was internationally/nationally/locally competitive with the scores weighted appropriately.
PATENTS – Beyond my area of expertise, but could be assigned a value in the same way as prizes/awards.
SPEAKER INVITATIONS – As for prizes/awards, speaker/chair invitations should be weighted based on location (international/national/local), with emphasis on any international speaker invitations.
SUPERVISION OF STUDENTS – A simple count of the number of active and completed Honours, Masters and PhD students an individual has/is supervising (0.25 for Honours, 0.5 for Masters and 1.0 for PhD?).
COMMUNITY ENGAGEMENT – This may be a qualitative variable not considered until the application reaches the panel review stage (that’s not to say this is not important, but rather speaks to the huge range in which individuals communicate/engage the community). Would be happy to hear suggestions about this.
DENOMINATOR VARIABLES
TIME – NH&MRC evaluates track record over five years, ARC uses ten years. Personally I prefer ten years, as it can demonstrate a sustained level of achievement, while also minimising the influence of a single bad year. This is also where relative to opportunity can be directly applied. If you are three years post-PhD then you could be evaluated over 3 years (or perhaps since the date of your first publication included above?).
CAREER INTERRUPTIONS – If you’ve had time out of research due to maternity leave, illness, carers leave etc. then this time can be deducted from the value above.
LAB CHANGE – One of the things that has always frustrated me is the disadvantage faced by early/mid-career researchers who might change labs 2-3+ times in the space of ten years before they establish themselves. In contrast, an established senior researcher has the advantage of not moving from lab-to-lab, and does not experience the same decrease in productivity for this (hopefully short) period of time. Therefore, I would propose a value (one month? two months?) for each time a researcher has changed labs (up to a maximum).
LAB SIZE – This is a difficult one, but the point I want to make here is that a senior researcher with ten post-docs would absolutely be expected to have a better track record than a new investigator with a single post-doc.
PREVIOUS FUNDING – On a similar note, a lab that has a track record of receiving a large amount of continuous funding will be expected to have a better track record than a lab with less funding.
CONCLUSIONS
This is by no means an exhaustive list, and is not meant to be. I simply want to push the conversation about how we can more accurately measure track record (relative to opportunity), and the much larger conversation about how to reduce the workload put on external assessors reviewing grants. Please let me know your thoughts in the comments below.
EDIT: I’ve included full definitions for my abbreviations
Research in Myogenesis 