The second SYoS workshop, which took place on 12 October 2023 at ETH Zurich, was dedicated to the topic of “reflexive bibliometrics”. The term “reflexive bibliometrics” may not be familiar to many, but it addresses an important phenomenon, namely, that many actors in the science system (e.g. Higher Education Institutions (HEIs) or individual researchers) adapt their behaviour when their performance is assessed using bibliometric indicators. For a definition of “reflexive bibliometrics”, see German Centre for Higher Education Research and Science Studies (DZHW): https://www.dzhw.eu/en/forschung/projekt?pr_id=605.
Workshop participants gathered in a collaborative atmosphere
23 people from 9 Swiss institutions* took part in the workshop following a lecture by Prof Emanuel Kulczycki on “Reframing Scientometrics: How ontological understanding of science influences what we count and how we interpret it“. The workshop was moderated by Dr Fabian Winter (University of Zurich).
Participants enjoyed the opening by Simon Willemin, who provided a concise overview of previous SYoS lecture on the “Opportunities and Challenges of Scientometrics” (speaker Dr Stephanie Hausman). For those who missed out on the June events, blogs summarizing the SYoS lecture and workshop are available online.
After Simon Willemin’s engaging kick-off to the workshop, participants joined “think tanks” of 5 to 6 people. The engaging discussion in small groups was striking. The think tank members collaborated on a series of tasks spread across three sessions: 1) Mapping the bibliometric landscape; 2) implications of the use of bibliometric indicators; 3) behavioural changes triggered by bibliometrics. The findings of the group discussions were presented in the plenum and produced a rich picture of insights, concerns, and solutions.
Mapping the Bibliometric Landscape: From Motives to Consequences
The first session posed a pressing question: What drives various actors in the scientific community — such as researchers, institutional managers, publishers, editors, or policymakers — to use bibliometrics for research assessment? Following intense discussions, each group showcased their findings to the plenum, fostering an enriching exchange of perspectives.
When investigating the motives for using bibliometric data in the different groups of stakeholders, it appeared that university managers and boards primarily turn to bibliometrics for comparative analyses. They appeared to be keen on understanding their university’s position in relation to other institutions. Beyond comparative endeavours, a push towards informed decision-making could be observed. Evidence derived from bibliometrics seems to pave the way for strategic, data-backed choices, from recruitment strategies to presenting precise solutions to governmental entities.
Researchers perceived bibliometrics as support tools for publication. Many seem to use them to gauge their professional standing and to compare their work with that of peers. One key aspect was the weight given to visibility: the brighter a researcher shines in the bibliometric spotlight, the greater is their intra and inter-disciplinarily recognition.
Publishers discerned bibliometrics as a compass directing the perceived worth of their journals, and informing their strategic moves.
For policymakers, bibliometrics seemed to be an essential tool for determining the return on investments in academia. Additionally, bibliometrics were viewed as indispensable in government agencies for guiding investment and funding decisions, given they provide simple and tangible evidence.
With similar enthusiasm, the second session steered the conversation towards the implications of the use of bibliometric indicators. How does the use of bibliometric indicators, especially for the purposes of research assessment, affect the modus operandi of researchers and the overarching strategies of their institutions? The insights from this session provided a clearer view of the influences of using of bibliometric indicators on actors within academia.
The discussions in this session also turned towards understanding the behavioural implications of bibliometrics. A tendency to ‘follow the metric’ could be observed, evident in a behavioural modification to fit into predefined metric moulds with the risk of potential misuse. This ‘follow the metric’-reaction seems common: Variation can be observed based on geographical and historical contexts, with nations like Poland, Switzerland, and the UK exhibiting specific responses. The session also highlighted related concerns, such as the rise of predatory publishers which might be a result of emphasising metrics. When used wisely, metrics could catalyse positive shifts, such as fostering transparent assessments and good research practices.
In the concluding session, the groups examined the behavioural changes triggered by bibliometrics. When are these changes beneficial or detrimental to researchers, HEIs, and other stakeholders in the academic ecosystem? And what precautions should be taken to ensure that bibliometric use neither undermines scientific values, nor deviates from the well-intended motives of developing bibliometrics in first place?
The discourse shifted towards a more holistic view of bibliometrics in this session. Re-occurring theme were the importance of values, the emphasis on academic freedom, and a nuanced view of complex issues. Amongst these discussions, the consensus that there is a need for advice on the responsible use of indicators and a requirement for guidance on the practical application of initiatives such as DORA and CoARA. Relying on bibliometrics for strategic moves only could be risky, but their structured objectivity can guide informed planning and strategic decisions in academia.
The above insights provide a first glimpse of the in-depth and multi-layered discussions that the workshop fostered and perspectives shared by the participants.
Bibliometric Paradoxes: Values, Behaviours, and the Quest for Meaningful Measurement
As with many academic discourses, the workshop was not free of controversial debates and unresolved questions. One key discussion centred on the essence of bibliometric indicators: Are bibliometrics about the values underpinning them, or is it crucial how we use the indicators?
An equally pressing question was how potential change could be achieved: Is it necessary to overhaul our entire system to bring about meaningful change? And if so, what is the best route for that transformation, a top-down or bottom-up approach?
Without getting too engrossed in methods and complexity of bibliometrics, a clear call for pragmatism was voiced. The academic community seeks clarity on foundational questions: What exactly do we aim to measure, and why is there an urgent need for measurement at all? Once this foundational work is done, attention can be paid to the ‘how’. If something becomes a measure, it must be a well thought-through measure.
The Journey Forward
While the unresolved questions leave us thoughtful and in demand of a deeper understanding, they also emphasise the importance and necessity of further exploration of the topic.
We are very pleased to offer another opportunity for discussion and learning. For those inspired by the discussions and keen to delve further into the world of bibliometrics, consider attending the 3rd SYoS lecture and workshop scheduled for 7 and 8 February 2024 in Lausanne. We are delighted to have Dr Elizabeth Gadd from Loughborough University as our speaker for the next SYoS lecture. Save the date and stay tuned for more exciting bibliometric discourses!
*The 9 Swiss institutions were: Bern University of Applied Sciences BFH, EPFL, ETH Board, ETH Zurich, FORS, University of Basel, University of St Gallen, University of Zurich and Zurich Universities of Applied Sciences and Arts ZHAW.
The author acknowledges helpful comments on the blog post by Dr Kathrin Thomas (University of Aberdeen).
