SYoS Lecture no 5

Assessing quality of research. New approaches in response to changing expectations.

The idea of accountability for research institutions receiving public funds based on the quality and impact of their research sparked already sixty years ago with the publication of de Sola Price's "Little Science, Big Science." The notion that publicly funded researchers should be held accountable for their work was a groundbreaking concept at the time. Today, this imperative has become a global reality, with governments, agencies, and institutional management bodies seeking to develop and implement methods and procedures to enhance and evaluate their research quality and impact.

The initial concerns surrounding research assessment focused on ensuring that publicly funded researchers were producing high-quality research. This involved developing quantitative methods for evaluating both productivity and quality of research outputs, using publication counts, citation metrics, and journal impact factors. These efforts aimed to provide a framework for assessing both the scientific production and the merit of research and for identifying areas for improvement. Field and discipline specific research practices were accounted for through field normalization and characteristic scores and scales. The role and effect of collaboration or self-citations was extensively documented.

However, as the scientific landscape has evolved alongside changing societal expectations, new challenges have emerged. The pace of scientific progress has accelerated, and the nature of research has become increasingly complex. In addition to these technical challenges, there is a growing recognition that research assessment must prioritize diversity, inclusiveness, and transparency. Researchers from diverse backgrounds are increasingly valued for contributing to the scientific community, bringing with them unique perspectives and experiences. However, this diversity also poses new challenges for research assessment, as evaluators must be able to recognize and value the contributions of researchers from underrepresented groups. Parallel to these societal changes, scientists themselves have advocated for the integration of additional or alternative approaches into the assessment toolbox in response to the identification of shortcomings and side-effects of established procedures.

These exogeneous and endogenous dynamic processes have crystalized into an altered and broadened understanding of the concept of quality. Consequently, contemporary practices in research assessment, rooted in the traditional perspective on research quality in terms of knowledge flows and reuse, are insufficiently geared towards capturing this broad spectrum of research characteristics.

In this lecture, I will delve into this altered concept of quality and its multi-facetted nature. In addition, the lecture will highlighting recent findings and explore emerging initiatives and research lines in research assessment that try to respond to those challenges while acknowledging the basic idea of researcher accountability.

Short Bio of Prof. Bart Thijs

Bart Thijs is a professor at KU Leuven, holding the ECOOM-funded chair in Bibliometrics and serving as a senior research fellow at the Centre for R&D Monitoring (ECOOM). He specializes in bibliometric research and provides bibliometric services to the Flemish government. He is also responsible for the design, development, and implementation of the data platform and research environment at ECOOM.

Prof. Thijs graduated in Psychology with a specialization in Statistics from KU Leuven in 1999. He spent several years in industry as a statistical consultant, focusing on the design and implementation of automated data analysis projects. In 2010, he earned his PhD in Social Sciences from Leiden University in the field of bibliometrics.

Currently, he teaches a course on the analysis of large-scale social networks within the Master of Artificial Intelligence program in the Department of Computer Science and a course on advanced research methods in quantitative science and technology studies in the doctoral program of the Department of Management, Strategy and Innovation (MSI) at KU Leuven. He is co-supervising PhDs at MSI and a joint PhD with the Faculty of A.

Prof. Thijs has co-authored over 70 publications, including journal articles, conference contributions, book chapters, and working papers. His research has evolved from profiling research institutes for improved bibliometric evaluation to exploring lexical and citation-based document similarities for science mapping. His current work focuses on graph representational learning and text embedding techniques using large language models to enhance science and technology mapping, document classification, and indicator development.

He has been an active contributor to numerous national and international research projects funded by agencies such as the EU, FWO-CELSA, and BMBF. In 2018-2019, he has been a visiting scholar at the Laboratoire d’Informatique de Grenoble, where he worked on text embeddings for canonical sections in full papers.