The third workshop of the Swiss Year of Scientometrics took place on February 8th, 2024, on the EPFL campus, and was divided into two streams. The “Open Science” stream addressed how to develop metrics for research assessment that consider the impact and value of Open Science outputs.
The workshop stream started off with a general discussion on the core principles and challenges inherent in this collaborative approach to research. Participants came from all backgrounds, from humanities to social and natural sciences and what came across immediately was the recognition that Open Science is not a monolithic concept. Rather, it is a dynamic framework that manifests differently across various disciplines, encompassing collaborative efforts, sharing commons, and ensuring research quality through best practices.
As we delved deeper into the discussion, it became evident that numerous obstacles stand in the way. Geographical barriers, technological limitations, and knowledge gaps all pose significant challenges. However, perhaps the most pressing concern is the sustainability dimension. We found ourselves questioning the balance between producing new data and responsibly utilizing existing resources, emphasizing the need for changing the research culture around data production, providing visibility to already existing data and rewarding sustainable practices.
Regarding metrics, the groups quickly came to the realization that, except for Open Access bibliometric indicators, no measure of Open Science was set up in their institutions. Moreover, it was agreed that, if such Open Science metrics were to be implemented at the scale of institutions, they would need to be adapted to fit the unique needs of different communities and disciplines. The participants collectively acknowledged the importance of ensuring that metrics do not incentivize quantity over quality and that they encompass both societal and academic impact. An idea was to design a general framework highlighting the different values of Open Science, such as transparency, societal impact, and sustainability along with the potential measurable metrics associated to these values. From this framework, disciplinary evaluation committees could decide which indicators to choose, based on the values they wish to reward within their communities. It was noted that imposing top-down metrics can perpetuate inequalities and incentivize gaming the system. Instead, engaging researchers in defining relevant metrics for their context fosters a more organic and meaningful approach.
Infrastructure was also a focal point of discussion, with participants highlighting the need for interconnected networks, persistent identifiers, and a focus on process-oriented best practices. The importance of keeping the data underlying this infrastructure as open as possible was highlighted, by using open-source data providing services such as OpenAlex rather than current commercial options.
While these insightful discussions mark a first step towards recognizing the gaps in our understanding of metrics related to Open Science and how to responsibly use them in the frame of research assessment, the journey is still long. As we navigate this path, it is imperative that these conversations continue to evolve. The forthcoming SYoS lecture and workshop coming up April 24th and 25th 2024, on “Openness of research information – Democratizing the use of scientometrics” by Ludo Waltman, will offer a great platform for further exploration and collaboration.
