This document describes the aims and methods to assess IN3 Research Groups for 2023-2026. This protocol is a revised version of the previous Evaluation Strategy for assessing research groups at IN3. This protocol aligns with the most recent developments in science and society regarding Academic Culture, Open Science, and international developments in research evaluation. It also aligns with the UOC’s strategic plans and most recent institutional agreements to reform research assessment (DORA, COARA).

The main goal of this protocol is to help evaluate research groups according to their aims and strategies. This protocol is committed to maintaining and improving research quality and social relevance and facilitating dialogue with the Scientific Advisory Board and the university's executive board.

This evaluation incorporates elements and methodology partially based on the work conducted by the Strategy Evaluation Protocol 2021-2027, a proposal developed by the Association of Universities in the Netherlands (VSNU), the Royal Netherlands Academy of Arts and Sciences (KNAW) and the Dutch Research Council (NOW). For more information about the Committee and Working Group composition, see https://www.universiteitenvannederland.nl/files/documenten/Domeinen/Onderzoek/SEP_2021-2027.pdf.

While efforts have been made to adapt and customise the content to suit the specific context of UOC, it is essential to acknowledge the foundational contributions of this Evaluation Strategy. Also, we thank Ismael Ràfols, a researcher at CWTS (Leiden University) and UNESCO Chair on Diversity and Inclusion in Global Science, for directing our attention to this strategy in the Netherlands.

This self-evaluation aims to shape a coherent narrative argument, supported by robust data, on the research aims, strategy, and results of the research group (RG). The evaluation period shall include, at least, the 5 past years of research activities. If the last evaluation of the group (or its inclusion in the Institute) occurred more than 5 years before the submission date, additional months or years should be included in the report.

Each group (those that must be assessed according to UOC’s quality assurance cycle) chooses the leading indicators and examples to provide evidence for the central assessment criteria: research quality, social relevance/impact, and viability. The strategy of the research group regarding the quality, social relevance/impact and viability of its research is the most central element in this self-evaluation. The indicators' relevance should be argued in the self-evaluation and based on the arguments it wants to develop. Other robust data may include benchmarking about peers, case studies highlighting the most distinctive and socially relevant accomplishments, and a SWOT analysis.

The self-evaluation should be limited to a maximum of 20 pages (appendices not included).

This report must be sent to the Scientific Advisory Board (SAB), which evaluates the RG based on the self-evaluation and an interview with the group leader and other potential RG members. The Scientific Advisory Board provides recommendations and future improvements for the RG, particularly its aims and strategy. The executive board of the university and research group will use this report as part of their quality assurance cycle.

This protocol aims to be a flexible instrument at the service of a productive and ongoing conversation about research quality, social relevance/impact, and future strategy of IN3 research groups.

The backbone of the self-evaluation is the strategy followed by the RG to achieve research quality, social relevance, and viability. Below, RG have some criteria to understand how these topics are defined and assessed:

The quality of the RG over the past 5-year period is assessed in its international, national or —where appropriate— regional context. The Scientific Advisory Board (SAB) does so by assessing an RG considering its aims and strategy. Central in this assessment are the contributions to the body of scientific knowledge. The SAB reflects on the quality and scientific relevance of the research. Moreover, the academic reputation and leadership within the field are assessed. The Board’s assessment is grounded in a narrative argument and supported by evidence of the scientific achievements of the RG in the context of the national or international research field, as appropriate to the specific claims made in the narrative. The protocol explicitly follows the guidelines of the San Francisco Declaration on Research Assessment (DORA) adopted by UOC.

The social relevance of the RG in terms of impact, public engagement and uptake of the RG is assessed in economic, social, cultural, educational or any other terms that may be relevant. Social impact may often take longer to become apparent. Social impact that became evident in the past 5 years may therefore well be due to research done by the RG long before. The Scientific Advisory Board reflects on social relevance by assessing the RG’s accomplishments considering its aims and strategy. The SAB also reflects, where applicable, on the teaching-research nexus. The assessment is grounded in a narrative argument that describes the key research findings and their implications. At the same time, it also includes evidence for the social relevance in terms of impact and engagement of the RG.

The extent to which the RG’s goals for the coming 4-5-year period remain scientifically and socially relevant is assessed. It is also assessed whether its aims and strategy, as well as the foresight of its leadership and its overall management, are optimal to attain these goals. Finally, it is assessed whether the plans and resources are adequate to implement this strategy. The SAB also reflects on the viability of the RG in relation to the expected developments in the field and social developments, as well as on the broader institutional context of the RG.

Aligned with institutional strategies, RG should also address at least the following five specific aspects. These specific aspects will also help to reinforce the narrative to explain the main three criteria as they are also integral aspects of each of the three major assessment criteria (Research quality, social relevance, and viability). These five aspects will contribute to illustrate how the RG organises and performs its research, how it is composed in terms of personnel and leadership, and how the RG is being run daily. These specific aspects are:

The SAB considers the extent to which the RG involves collaboration and cooperation between different disciplines -which can include also different perspectives, methodologies, and knowledge- in the preparation and execution of its aims and strategy. It also considers to which extent the RG opens and collaborates with other researchers and research groups, particularly from IN3. Furthermore, the SAB considers whether the RC actively promotes interdisciplinary teams to carry out joint research projects, institutionally or at the interinstitutional level. In the self-evaluation, the RG reflects on the role of interdisciplinarity in its research and the importance of synergies, dialogues, and collaborations to other disciplines -and perspectives, methodologies, and knowledge- to it research strategy.

The SAB considers the extent to which the RG involves stakeholders, if possible and relevant, in the preparation and execution of the aims and strategy. It also considers to which extent the RG opens its work to other researchers and social stakeholders in its strategy and policy context. Furthermore, the committee considers whether the RG reuses data where possible; how it stores the research data according to the FAIR4 principles; how it makes its research data, methods, and materials available; and when publications are available through open access. Even if Open Science was not yet considered by the research unit for the past period, the assessment committee evaluates the RG’s considerations and plans about Open Science. In the self-evaluation, the RG reflects on how it involves stakeholders, to which extent the RG opens its work to other researchers and social stakeholders, how it pays attention to other aspects of open science and what its plans are in this respect.

The SAB considers the supervision and instruction of PhD candidates, including PhD education at relevant institutional graduate school(s) and (national) research school(s), according to their aims, strategy, and policy. Furthermore, the Board considers whether the quality assurance system is functioning properly. Here, too, the goals that the RG has set for itself are important (PhD training, mentoring, and coaching). In the self-evaluation, the RG reflects on the institutional context of the PhD programmes, the PhD programme content and structure, quality assurance, the selection and admission procedures for PhD candidates, as well as the position of PhD candidates and PhD training in the group’s research. Furthermore, the RG reflects on the supervision of PhD candidates, the effectiveness of the Training and Supervision Plans, the guidance of PhD candidates towards the job market, duration, success rate, exit numbers and career prospects for PhD candidates.

Openness, (social) safety and inclusivity: The SAB considers the openness, (social) safety and inclusivity of the research environment. In the self-evaluation, the RG reflects on its culture in terms of appreciating the multiplicity of perspectives and identities in the workplace; on which measures are taken to ensure openness, safety, and inclusivity, and on how responsibility is taken by leaders of and within the RG to contribute to such an academic culture. Gender aspects must also be included in this section. Research integrity: The assessment committee considers the RG’s policy on research integrity and how the group facilitates the relevant actions and requirements formulated in the European code of Conduct for

Research Integrity: In the self-evaluation, the RG reflects on data integrity and the extent to which an independent and critical pursuit of science is made possible within the group. Furthermore, the RG reflects on the degree of attention given to integrity and ethics, on the prevailing research culture and mode of interaction, as well as on relevant dilemmas that have arisen and on how the RG has dealt with them. These dilemmas could include issues related to authorship, ethical considerations regarding privacy or collaborations with stakeholders.

Diversity: The SAB considers to which extent diversity (including gender, age, ethnic and cultural background, and disciplines) is a concern while it also evaluates the actions and plans for the future of the RG. In the self-evaluation, the group reflects on where it stands at present concerning diversity in relation to its aims, strategy, and policy. Furthermore, the RG reflects on how it guarantees diversity-promoting HR practices such as inclusive selection and appraisal procedures.

Talent Management: The SAB considers the RG’s policies on talent selection and development in relation to its aims and strategy. More specifically, it evaluates the group’s recruitment policies, opportunities for training and development, coaching and mentoring, and career perspectives for researchers and research support staff in different phases of their careers. In the self-evaluation, the RG reflects on its selection, training, promotion, and retention policy and on how it offers opportunities for diverse career paths. This reflection includes considering how the RG ensures that researchers are properly evaluated, rewarded, and incentivised.