IMPACT! Academics, Citation, and Scholarly Self Delusion

Greetings boys and girls, in today’s lesson I want to talk about “citation impact factor.” And just what is citation impact factor you ask? Well, impact factor is what scientists use to assess their “impact” on things. It is a measure, I suppose, of a scientist’s “value” as scientist. Despite some recent attempts to kick it up a notch, citation impact is a very simple measure. You write an article, you publish it, and you count how many other scientists “cite” your article (i.e. include it in the reference section of their article or book). That’s all there is to it. So for example if I write an article on labour process theory, and if I publish it in the journal, Work, Employment, and Society (Sosteric, 1996), and if that article goes on (for a time) to become one of the top ten most cited articles in the journal (as it did), perhaps gathering as many as (gasp) 99 citations, then it is judged to be a high impact article.

Yay!

If its high impact I get to point to it and say “I wrote something important.” If its not high impact then all I can do is hang my head in shameful scholarly obscurity, presumably confined to the dustbins of this planet’s scientific history.

Yawn.

As boring as it is, it seems sensible and logical enough, if you don’t know any better, but I think there are problems. For example, if you unpack the notion of “impact” a bit and ask “impact on what,” the validity of the whole enterprise comes into question. I mean, what can you really say about an article that has been cited 99 times? Does that mean it has been read 99 times? Hardly! Many scholars cite articles based on only a quick read of the abstract, and many more read things that they never cite at all. I mean, I’ve read thousands of books and articles in my life time and I’ve cited only a handful. Try as we might, it hardly seems right, or fair, to suggest that “number of citations” means anything at all in those circumstances. And these are just the most obvious problems. Even if you get past this rather glaring “fail” to the point where you would suggest that having a 99 citation count actually means something , you still have to answer the questions “means what?” And that’s not so easy to answer either. Does having a 99 mean your ideas are part of the intellectual fabric of this planet? Maybe! But then again, maybe not. You can’t tell that from the number alone. In order to determine actual impact you would have to do some complicated (and expensive) qualitative research, and if you did I’m pretty sure you’d find that sometimes an idea is incorporated into the fabric after 99 citations, and sometimes not. In other words if you did the research you’d find the citation measure is unreliable, and invalid. And as any methodologist will tell you, if a measure is unreliable and invalid, it’s totally and utterly useless.

Even what I’ve said so far is enough to call into question the validity of citation counting, but some people might want to grasp and claw at the measure anyway. There is, however, a final consideration that makes it clear just how ridiculous citation counting is and that is the fact that sometimes scientific ideas can have major impact without ever getting cited at all. Consider this. I was heavily influenced by the thinking of Karl Marx, but I don’t cite him in every article I write. I don’t feel I have to, but his influence is there all the time. And if that doesn’t cause you to call into question the citation analyst’s little radar, consider this. A few years ago I wrote an article on forms of emotional abuse commonly found in schools (Sosteric, 2012). In the time that it has been on this site many people have commented on it, and many hundreds of thousands of people (parents and teachers) have read it. Just last year it was picked up by Educational Testing Service in the U.S.A to be used in their teacher certification materials. The contract I signed specified 50,000 teachers over the next ten years! This means that my ideas on emotional abuse in schools will be influencing the next generation of teachers as they complete their programs and pass certification. Clearly this article on abuse in schools is having a major impact, but AFAIK it hasn’t been cited once.

So where does all this leave citation impact factor? If you ask me, it is an unreliable, invalid, ridiculous, and useless measure. And frankly, I’m not the only one to say so (Adam, 2002; Baum, 2013; “Not-so-deep impact,” 2005; “The ratings game,” 2010). When you start to get really deep, the issues keep piling higher. Citation counts may give the university administrator something to point to when they’re looking at you for a promotion (which to be honest is what they are really all about), but other than that they have no utility at all. You won’t get a a fair representation of your impact by a citation count, so don’t even bother to try. You also shouldn’t use it to prove just how important you are, because its a vacuous and meaningless measure that speaks more to academic ego than it does to actual contribution. If you want my advice, just give it up.

References

Adam, D. (2002). The counting house. Nature, 415(6873), 726-729. doi: http://dx.doi.org/10.1038/415726a

Baum, J. A. C. (2013). The excess-tail ratio: correcting journal impact factors for citation distributions. [Article]. M@n@gement, 16(5), 697-706.

Not-so-deep impact. (2005). [10.1038/4351003b]. Nature, 435(7045), 1003-1004.

The ratings game. (2010). [10.1038/464007b]. Nature, 464(7285), 7-8.

Sosteric, M. (1996). Subjectivity and the Labour Process: A Case Study in the Food and Beverage Industry. Work, Employment, and Society, 10(2).

Sosteric, M. (2012). The emotional abuse of our children: Teachers, schools, and the sanctioned violence of our modern institutions. The Socjournal, March.

Cite This Article

Dr. S. (2014). IMPACT! Academics, Citation, and Scholarly Self Delusion. The Socjourn. [http://www.sociology.org/citation-impact-factor-a-scholars-cautionary-tale/]

By: William T. Hathaway

SUMMER SNOW is a spiritual novel set now amidst the war on terrorism as an American warrior falls in love with a Sufi mystic and learns from her an alternative to the military mentality. As US Special Forces battle al-Qaeda, the escalating violence threatens their future together and the lives of thousands in her country. To save them, she shows him an ancient transcendental way to bring peace to the collective consciousness and prevent terrorism. But can they make it work in time? A story of love in the shadow of destruction, SUMMER SNOW blends passion, adventure, and mystic wisdom to convey its theme that higher consciousness is more effective than violence and that women may be more able than men to lead us there

[ Amazon.com | Book Finder ]