No one can read it all
Half a century ago, it was possible for a researcher to keep abreast of the literature in their field. One could browse through most of the journals within the discipline, and probably a few outside, with a reasonable investment in time. But circumstances have changed. The world of publishing has seen tremendous growth and there are currently many more journals for each discipline. At the same time, research has evolved to become more interdisciplinary. New areas of research with their own publications developed and existing niches expanded into distinct dominions; both further increasing the number of potentially important journal outlets for researchers. Today, it is almost impossible to keep track of what is happening everywhere. No one can read everything. Choices must be made.
Clearly, the abovementioned is one of the driving factors why people use the impact factor (IF) as a heuristic to help decide which journals to read and to publish in. At this point, it needs to be reiterated that the impact factor was first introduced to facilitate the choice of which journals to subscribe to. It served to answer the question “On what journal subscriptions is the library coin well spent?” and not “What is a good way to measure the excellence of a journal?” Bearing this contrast in mind, this is a good point in time to remember Goodheart’s Law [2]: “When a measure becomes a target, it ceases to be a good measure”. Once the impact factor became a target, it started a problematic game.
What makes a social game? First, players are needed. In our case, these are publishers, editors and scientists. Secondly, there must be a metric to decide who is awarded points, or even who is crowned winner of the game. Obviously, here, the impact factor serves as a metric. Finally, there must be rules that everyone needs to adhere to and that are enforced. Historically, the scientific community has relied on the concept of scientific integrity to fill the role of rules. And as far as enforcement is concerned … much potential still remains. Unsurprisingly, this foundation has inspired a lot of creative behaviors, and these behaviors can be highly rewarding, as far as careers go [4].
Self-citations are a normal phenomenon. Any journal will normally self-cite to some degree, because new work may build on previous work in the same journal. Typically, whenever impact factors are wielded for advertising purposes, be it by journals, be it by authors highlighting where they managed to publish, these impact factors contain self-citations and are thus inflated. Just for clarity’s sake, impact factors without self-citations have been available for decades. But it is customary to use the other ones because a higher number is more impressive. In the publishing world, this custom can be used by journal editors to play the game a bit better. A higher impact factor of a journal will attract better papers, will attract more citations, will increase the IF. It’s a wonderful self-perpetuating cycle. You just need to find bootstraps to get the first step going. And how would you do that?
Let’s imagine, you take a position as an editor-in-chief of a journal that has an IF of below 0.1. This is more than no impact factor at all [7], but is tremendously low, belonging to the bottom 3% of the ranking. Your mission is to improve the IF to improve the status and reputation of the journal. The simplest way would be to introduce an editorial, that relates articles in the current issue to articles from the previous two years, of course with a reference to each article. Ideally, on average each article receives a citation and one citation per article already makes an IF of around 1 (depending on the editorial counting as an article or not). Suddenly you move from the 3rd percentile in the ranking to somewhere around the 63rd, which is a huge increase. Remember, in part 2 of the series, we saw that an impact factor of 4 and below already includes 85% of the journals listed in this index.
Alternatively, and crossing yet another boundary, an editor could make it part of their job to actively solicit self-citations. For example, citations to any articles (from the previous two years) remotely connected to a submitted article could be imposed on authors. Very quickly, a number of valuable citations can be garnered.
The two ways to self-inflate described above are obvious modes to increase one’s citations. However, they are also easily detected. The discrepancy between impact factor with and without self-citations is monitored and if it reaches a certain level [9], a journal will be scrutinized. Journals may receive a one-year time-out from impact factor as a penalty [10], but not always there will be consequences [11].
Now let’s look at a more elaborate strategy to get that magic number up. It’s harder to detect, but a partner-in-crime is needed: The citation duo.
The next step up the ladder is to collaborate in a larger group and to cite each other’s work regularly. Cite me mine, I’ll cite you yours. The origin of citations is distributed on several shoulders and becomes harder to detect. This is called a citation cartel, and it facilitates skewing the impact of several journals or researchers simultaneously. And of course, is not a very welcome behavior. Clarivate will crack down on this once it’s noticed and journals will end up excluded from the ranking.
Conclusion
To sum up, we caught a quick glimpse on a few of the ways to play the impact factor game. There are many more, some of them obvious, some that will be so creative that we still cannot fathom what they might be. Playing the game well promises substantial rewards. The metric is very clear, but not the rules.
Clarivate has internal rules in place that serve as basis for investigations. These rules, however, are not officially communicated and can only be inferred retrospectively. It is problematic, that the ultimate judgement of what is accepted or not resides in the hand of a private company, and additionally, that there is no transparency. The scientific community who is affected, does not have a say.
The game will be played, if we want or not, so the issue needs to be addressed. On the other hand, unclear rules might deter some people of playing. If the rules are crystal clear, the game will be played out to the maximum. For example, tax optimization mostly stays within the rules of the law and no-one would say this is a behavior that is positive for society as a whole. A comparable behavior is diving in football. The benefits outweigh the consequences of getting caught. But it comes at the price of making football more annoying and boring to watch as a viewer. Having well-defined boundaries for citation behavior may give more power to the community, but it might not be a solution to the underlying problem.
What are the consequences of getting caught in the citation business? Self-citations are normal to some degree, but problematic when excessive. Authors do not have much to lose when engaging. Journals might receive a time-out from the impact factor – a slap on the wrist. Retraction watch has its own category “citation manipulation”: It contains 45 posts, over a period of 14 years [14]. This highlights the fact that the consequences of playing the citation game to the border of an ill-defined legality definitely outweighs any ramifications one might suffer. That needs to change!
This is part 3 of the series “Shananigans with Impact Factors”. Read also:
[1] Image: pexels.com
[2] https://en.wikipedia.org/wiki/Goodhart%27s_law
[3] Image: www.pxfuel.com
[4] In his blog post Richard Phelps nicely illustrates what fruits of playing the game can be harvested over time: https://retractionwatch.com/2022/05/17/how-citation-cartels-give-strategic-scholars-an-advantage-a-simple-model/
[5] Image by Suzy Hazelwood: https://www.pexels.com/de-de/foto/hotrod-druckgussmodell-an-bord-1422673/
[6] Image: https://pixabay.com/photos/peacock-feather-bird-peafowl-3098451/
[7] On https://jcr.clarivate.com/ a total of 21’848 journals were listed for 2023, but only 20’787 of them had an impact factor. Clarivate announced for the 2023 release, for the first time, all journals in the Web of Science Core Collection had Journal Impact Factors. https://clarivate.com/news/clarivate-announces-changes-to-the-2023-journal-citation-reports/
[8] Image: https://imageresizer.com/meme-generator/edit/obama-giving-obama-award
[9] Self-citation is acceptable to some degree and in the past, an attempt was made, to estimate what the threshold is. https://scholarlykitchen.sspnet.org/2017/06/05/reverse-engineering-jcrs-self-citation-citation-stacking-thresholds/ Most probably, this limit will have changed in the seven years since.
[10] https://scholarlykitchen.sspnet.org/2011/10/17/gaming-the-impact-factor-puts-journal-in-time-out/
[11] https://retractionwatch.com/2017/06/19/journal-hit-citation-scandal-named-among-top-field/
[12] Image: https://www.reddit.com/r/MemeRestoration/comments/eq1jwh/i_restored_in_hd_4k_the_original_spiderman/
[13] Image: https://imgflip.com/memegenerator/341160306/Spiderman-Circle
[14] https://retractionwatch.com/category/by-reason-for-retraction/citation-manipulation/
