One of the interesting aspects of scientists in academia is that they love to quantify things wherever possible.  In some circumstances, this is a very good idea, as it can lead to greater precision and rigor.   But the increased rigor can come at the cost of decreased information content, and one can end up quantifying, very precisely, only minor themes of a larger story.  Such is the case in the very difficult enterprise of judging the value of one of your colleagues.

When professors review their colleagues, they often judge their colleagues’ research based upon three numbers and only three numbers:  (1) the number of peer-reviewed publications, (2) the number of times said publications have been cited, and (3) the amount of grant support obtained.  Much could and has been said about whether or not these are appropriate sorts of beans to count, but there is no doubt that these three kinds of beans only capture a small part of the value of a faculty member.  More about that later — in this post I want to focus in particular on counting the number of peer-reviewed publications, in particular on one very curious aspect of it: that 1 can equal 5.

Imagine that J. Doe spent a single year producing a single publication, and that J. Doe is the sole author on that magnificent publication.  She puts that publication down on her CV:

Doe, J. 2013. My paper. Journal of Many Papers, 42:1-50.

When it is time to judge J. Doe, this part of her scholarly productivity is viewed as worthy of one bean.

But now imagine another researcher, J. Ray, who took a different tactic.  She decided to spend 1/5 of a year coauthoring a paper with 4 other researchers. At the end of the year her CV looks like this:

Ray, J., Me, A., Fa, B., So, C., La, D. 2013. Our paper. Journal of Many Papers, 42:51-100.

Now one might naively expect that J. Ray will have earned exactly 1/5 of a bean for her efforts, but that is not how the system works: J. Ray will get a whole bean for this paper.   And, in their home institutions, each of the other authors will be viewed as having earned exactly one bean too, so the total number of credit beans awarded for that single publication is five.

But if the papers are of equivalent quality and importance, how does it make sense to award only one bean for the first paper and five total beans for the second paper?  It makes one think that J. Doe should have submitted her paper with an author list as follows:

Doe, J., Doe, J., Doe, J., Doe, J., and Doe, J. 2013. Our paper. Journal of Many Papers, 42:1-50.

One could argue that J. Ray is awarded with a whole bean for being collaborative, but if that is the sole quality that is valued (which it shouldn’t be), then J. Doe would be given zero beans, which is obviously silly.  So what quality is being valued?  The total amount of product created by the efforts of J. Doe?  In that case, J. Doe should surely get more beans than J. Ray, as she did a whole paper but J. Ray contributed 1/5 of a paper.  But that is too simplistic as well.

If academics want to judge people rigorously, then there should be more rigor in the discussions of values and how to quantify them.  Otherwise, bean-counting is not worth a hill of beans.