On November, 13, 2024, the University of Fribourg hosted the 5th lecture of the Swiss Year of Scientometrics: Prof. Bart Thijs from the KU Leuven spoke on “Assessing quality of research: New approaches in response to changing expectations”. The video and slides of the talk are now available.
The event took place in the lecture hall of the architecturally impressive Miséricorde main building. Dr. Katja Wirth, Head of Research Promotion Service at the University of Fribourg, welcomed the guest speaker and audience. Thirty-five participants attended the talk on a grey November day.
After her welcoming notes, Katja Wirth handed over to Prof. Bernard Ries, Vice-Rector of the University of Fribourg, for a welcome address. Bernard Ries emphasized the importance of a balanced approach to research assessment and a well-thought-through use of metrics.
Prof. Bart Thijs began the lecture by comparing research evaluation to building a house: as it takes more than just bricks to build a house, achieving greater societal benefit from the science system requires more than just scientific knowledge, he suggested, as science evaluation involved relying on experts and making numerous decisions. This process was often outsourced and carried out through research assessment, intertwined with the process of ensuring quality.
Bart Thijs elaborates that evaluators sought to identify potentially usable contributions. This was an ex-ante decision-making problem that could be addressed by examining the strategy (e.g., publications in peer-reviewed venues as an indicator of potential quality), the impact (e.g., citations as an indicator that a contribution has been used), and the production (e.g., the number of publications) of scientists or research groups.
Further, Bart Thijs continued, the decision-making process could be structured within a matrix of pre-defined criteria, in which potentially usable contributions could be identified through metrics for strategy, impact, and production. Yet, he also suggested that it was important to remember that the data underlying these metrics could contain errors and biases. Therefore, the system was not perfect, but it required continuous efforts to improve it. Quality, according to Bart Thijs, was also negotiable and community-dependent, and, thus, required ongoing discussions about the relevance of the underlying metrics.
Moreover, the utility of science should not be confined to three dimensions (i.e., strategy, impact, production), but rather address global and societal challenges that, today, required the inclusion of additional dimensions, such as international collaboration, interdisciplinarity, open science practices, and contributions to the UN sustainable development goals. After discussing the relevance of existing indicators for measuring these dimensions and critically reflecting that some of these might provide less meaningful information than previously thought, Bart Thijs addressed unintended biases or side effects in decision-making processes.
Any decision-making process, so Bart Thijs, had unintended biases and side effects in any assessment, whether quantitative or other indicators were applied. However, the (mis-)use of indicators, such as Journal Impact Factor (JIF), could be harmful. In addition, some indicators were also susceptible to gaming, which, in return, could lead to unwanted side effects. For instance, established actors were often favoured over newcomers, and gender biases could be perpetuated. All of the above, could also have a reverse effect on mental well-being of the researchers, which, according to Bart Thijs, should not be underestimated.
Bart Thijs concluded by re-emphasising that the framework of science involves different stakeholders, including decision-makers and scientists and expressed the hope that, just like the bricks of a house, the scientific knowledge will enable us to build a world that is prosperous for everyone. Research assessment should contribute to this vision, whatever form it might take.
At the end of the event, Dr David Johann, Head of the SYoS project, thanked the University of Fribourg for hosting the event and for the excellent collaboration in preparing for it. David Johann also mentioned the Swiss Open Academic Data (SOAD) initiative of EPFL and ETH Zurich aimed at building a community for Swiss researchers and practitioners around open academic data. Finally, he announced that further SYoS events were planned and would take place at ETH Zurich in September 2025.
All participants were able to discuss the talk with Bart Thijs and the SYoS team over an apéro after the lecture.
The slides are available for download on Zenodo in the SYoS community.
The video recording of the lecture is available:
00:00:00 Introduction by Dr Katja Wirth
00:02:08 Welcome address by Professor Bernard Ries
00:07:37 Start of the lecture by Professor Bart Thijs
01:17:27 Dr Katja Wirth opens the Q&A session
01:28:32 Outlook and closing remarks by Dr David Johann
01:30:58 End of the event
The author acknowledges helpful comments on the blog post by Dr Kathrin Thomas (University of Aberdeen).
