A quantitative analysis of track record?

Yesterday The Conversation published an article titled “Science funding should go to people, not projects” which has as its leading hypothesis that we should shift from a project funding model to a person funding model. The reason for this is that it is difficult to compare the relative merits of projects from different fields, and in times of extremely low success rates leads to the preferential funding of “safe” projects (often projects which are essentially complete already). Immediately following publication of this article, the twitterverse exploded, with several very interesting reply threads arising (unfortunately the discussion became a bit ad-hoc, and no # was devised at the time, however @Dr_Mel_Thomson has created a great Storify page).

Let me say straight out that I am against funding people in the absence of projects. My concern (similar to those raised by several others) is it unfairly biases against young Early and Mid-career Researchers (EMCRs) and women. It encourages funding to go to senior academics in large established labs. To counter this, many will argue that track record will be judged ‘relative to opportunity’, and this leads me to the topic of this post, can we shift away from the current qualitative measure of track record to a more quantitative metric that more accurately reflects the idea of ‘relative to opportunity’?

With the rebuttal period for the major funding rounds for the Australian National Health & Medical Research Council (NH&MRC) and Australian Research Council (ARC) just about to finish, I’m sure everyone has their own anecdotal story to tell about the evaluation of track record. From a personal perspective, I’ve had comments suggesting I had a “moderate” track record from one assessor, and an “outstanding” track record from another – on the same grant. Regardless of what my track record may or may not be, these comments indicate the problems associated with a qualitative assessment of track record.

Assessors for NH&MRC are told to ignore journal impact factors and other metrics such as H-index when assessing track record, and they are told to assess the candidate relative to opportunity. In the absence of guidelines, this becomes haphazard at best (through no fault of the assessors, having assessed several grants over the last three years I find it tough). While it is easy to pick the investigators/teams with the absolute top track records (6/7’s for NH&MRC and A’s for ARC), and the non-competitive track records, the next tier down becomes difficult (think 5’s for NH&MRC.and B/C’s for ARC).

Imagine logging on to RMS (for ARC) or RGMS (for NH&MRC) and being able to quickly calculate a researcher score (lets go with the current NH&MRC system of 1-7). If your score comes out as less than 4.5 you are ineligible to submit a grant as a lead investigator. Straight away, this would reduce the burden placed on external reviewers by triaging applicants who are deemed not-competitive. A score of 4.5+ allows you to submit your project proposal for review. During the rejoinder process, RGMS/RMS allows you to update your track record and recalculate your score, at this point a minimum of 5.5 is required to progress. This would allow those researchers who have that big, exciting paper published in May/June to significantly improve their score.

Following these two rounds of culling based on track record, the remaining grants could go to the grant review panels (NH&MRC) or College of Experts (ARC) for discussion. Track record scores could then be adjusted by the respective panels when deemed necessary.

Now the big question, what variables do we plug in to generate the magical score? Can we limit the potential for researchers to game the system? Can we make the quantitative score fair across a broad range of disciplines? Importantly, any quantitative value/metric MUST improve on the current system. This is where I make my disclaimer, these variables are simply a list of things I would like to see incorporated, it is by no means exhaustive and there will be plenty I have forgotten (feel free to point out things I’ve missed in the comments), but these are things I believe are important. Any score should be a combination of both track record (numerator) and time/opportunity (denominator).


FIRST/LAST AUTHOR PUBLICATIONS – First and foremost, any track record score needs to take in to account both the quantity and quality of publications. Generating a fair score for this is difficult, impact factors can bias against certain smaller fields, citations are not necessarily a true indication of the impact of a piece of work, Altmetrics can be gamed (to some extent). A significant improvement in recent times are the inclusion of article level metrics (how many times an article has been viewed, downloaded, cited). Each one of these variables is imperfect on its own, but a combination of journal impact factor, article level metrics, and online impact (altmetrics) could give a reasonable measure of the “impact” of the work.

SECONDARY AUTHOR PUBLICATIONS – The number of first/last author manuscripts are particularly important, but we also need to recognise the importance of secondary author manuscripts as well. Science is supposed to be collaborative, and the last thing we want to do is encourage researchers to work in isolation, and yet so many reviewers count up the number of first and last author manuscripts and ignore everything else. This removes the impetus for collaboration. Could the publication metric be weighed by position in the author list? (1.0 for first/last, 0.2 for secondary author?).

PREVIOUS FUNDING – As you’ll see, I’ve included this in the calculation of the NUMERATOR and the DENOMINATOR below, as funding often begets more funding. But it is important to recognise a track record of being able to successfully obtain funding, from national funding agencies, international sources and industry.

PRIZES/AWARDS – At the moment I’m not convinced this section really adds much to the track record score, as assessors tend to focus on publications above all else. But prizes and awards are important, they are often awarded by our peers in recognition of outstanding work. Again, difficult (but not impossible) to put in to a numerical score. Any scoring system should take in to account whether the prize/award was internationally/nationally/locally competitive with the scores weighted appropriately.

PATENTS – Beyond my area of expertise, but could be assigned a value in the same way as prizes/awards.

SPEAKER INVITATIONS – As for prizes/awards, speaker/chair invitations should be weighted based on location (international/national/local), with emphasis on any international speaker invitations.

SUPERVISION OF STUDENTS – A simple count of the number of active and completed Honours, Masters and PhD students an individual has/is supervising (0.25 for Honours, 0.5 for Masters and 1.0 for PhD?).

COMMUNITY ENGAGEMENT – This may be a qualitative variable not considered until the application reaches the panel review stage (that’s not to say this is not important, but rather speaks to the huge range in which individuals communicate/engage the community). Would be happy to hear suggestions about this.


TIME – NH&MRC evaluates track record over five years, ARC uses ten years. Personally I prefer ten years, as it can demonstrate a sustained level of achievement, while also minimising the influence of a single bad year. This is also where relative to opportunity can be directly applied. If you are three years post-PhD then you could be evaluated over 3 years (or perhaps since the date of your first publication included above?).

CAREER INTERRUPTIONS – If you’ve had time out of research due to maternity leave, illness, carers leave etc. then this time can be deducted from the value above.

LAB CHANGE – One of the things that has always frustrated me is the disadvantage faced by early/mid-career researchers who might change labs 2-3+ times in the space of ten years before they establish themselves. In contrast, an established senior researcher has the advantage of not moving from lab-to-lab, and does not experience the same decrease in productivity for this (hopefully short) period of time. Therefore, I would propose a value (one month? two months?) for each time a researcher has changed labs (up to a maximum).

LAB SIZE – This is a difficult one, but the point I want to make here is that a senior researcher with ten post-docs would absolutely be expected to have a better track record than a new investigator with a single post-doc.

PREVIOUS FUNDING – On a similar note, a lab that has a track record of receiving a large amount of continuous funding will be expected to have a better track record than a lab with less funding.


This is by no means an exhaustive list, and is not meant to be. I simply want to push the conversation about how we can more accurately measure track record (relative to opportunity), and the much larger conversation about how to reduce the workload put on external assessors reviewing grants. Please let me know your thoughts in the comments below.

EDIT: I’ve included full definitions for my abbreviations


Effects of GFP on C2C12 metabolism

I wanted to share some results we recently generated on the Seahorse Bioanalyzer, looking at the effect of GFP on cell metabolism. I was really surprised at the size of the response, and how reproducible it was. Hopefully others will find this data useful/interesting when interpreting their own results. Please let me know in the comments section below if you have found something similar.

This year my BSc(Hons) student (the very talented Ms Alex Webster) has been working on a C2C12 cell based project involving transient overexpression of a GFP tagged protein. Alex has completed several experiments so far looking at the effect of her protein on rates of proliferation and differentiation. Because of my ongoing interest in cell metabolism, Alex also used the Seahorse XF24 Bioanalyzer to examine the effect of her protein on OxPhos and glycolysis. She compared metabolism across three groups, control, GFP alone and her protein tagged with GFP, the results of the first two groups are presented below (we are planning on publishing a study with the tagged protein, so it isn’t included).

GFP_bioenergeticsMETHODS: The two graphs are from a single mitochondrial stress-test, run on a Seahorse XF24(3) machine, measuring oxygen consumption rate (OCR, top graph), and extracellular acidification rate (ECAR, bottom graph). In each graph, the green line/points indicate the GFP group, and black the control. Cells were seeded at a density of 25,000 cells/well and transfected with a commercially available GFP plasmid from Origene (pCMV6-AC-GFP) using Lipofectamine3000. The assay was run the next day as previously described by Dr David Nicholls (I will provide a full outline of our methodology in the ‘Muscle Methods’ section at a later date).

RESULTS: As you can see, overexpression of GFP in C2C12 cells results in a significant 3-fold increase in basal glycolysis (bottom graph) and a minor decrease in maximal oxidative capacity (peak observed between points B and C in top graph). Alex has repeated this experiment several times, and in every case experiment she has observed a 2-3 fold increase in glycolysis.

These results have important implications for the use of fluorescent tagging in cell culture studies. Altered cellular metabolism can lead to altered metabolite levels which can directly influence the transcriptional profile of the cell. Importantly, these results also indicate the need to confirm the metabolism of cells isolated in lineage tracing models (such as the often used ROSA26eYFP mouse).

As we have only used a single plasmid at this stage, I would be really interested to hear what experiences others have had when using fluorescent tags and the Seahorse.