Progress

Currently, the group engages in five on-going projects:

  • Incorporating Intangible Information for Assessment Optimisation

The idea arose from the discussion at the 2017 GYA Annual general Meeting within the working group and was preliminary implemented in 2018. We prepared an anonymous relational database of the applicants and the reviewers with the gender, nationality, and discipline attributes. We also designed a short questionnaire to be filled out during the member selection process 2018 and 2019 by the selection committee members. The questionnaires were collected and preliminary analysis was performed. We did few overall preliminary analyses and planning to perform further analysis using the 2018 and 2019 applications. The ultimate aim is to initiate a new project for identifying novel factors which reflect the implicit personal qualities of a person which are yet vital professionally.  Read more here.

  • Analyzing Global Views of Scientific Excellence

Koen Vermeir spearheaded and led the first global study of research evaluation practices since 2016. We need this data-informed approach to be able to propose evidence-based recommendations that can improve current evaluation practices and research culture as a whole. In order to have a comparable data set on a global scale, we started the Analysing of Global Views of Institutional Promotion Criteria, focusing on criteria to become a full professor. The first results and a provisional report of this project was presented at the AGM 2018. By now, we have come out with an analysis from 80 documents we obtained across many countries. More documents will be further analysed. We expect to publish a report in the end of 2019.

  • Assessment, Publishing Models, and Open Science

The group is looking into following up on the 2018 report on “Publishing Models, Assessment, and Open Science“.  There are some questions in the problem framework that are worth exploring further.

  • Study on Reproducibility and Replicability in Science and Engineering

In the spring of 2018, the working group was invited by the United States National Academies of Sciences, Engineering, and Medicine to provide expert input on the factors that affect reproducibility and replication of data produced from Science and Engineering research. A survey of GYA members provided input for a presentation in May 2018, which provided a set of comments and recommendations on replicability, based on GYA members’ input and on their personal expertise. A report on “Young Scientist Perspectives on Replicability and Reproducibility in Science and Engineering” is forthcoming.

  • IAP Research Excellence Study: “Strengthening Research Evaluation: A Global Assessment and Dialogue”

Koen Vermier was nominated to join the InterAcademy Partnership working group on “Strengthening Research Evaluation: A Global Assessment and Dialogue”.

The Scientific Excellence Working Group continues to work together with the InterAcademy Partnership (IAP), ICS, ALLEA, the JRC, and several European Commission Expert Groups to drive change in the perception and measurement of scientific excellence. These collaborations have led to significant results that are impacting policy on a local, regional as well as global level (see the section “outcomes”).
Currently, we are working with IAP on a proposal on “Evaluating Research: Assessment and Improvement of Current Practice” and we are collaborating with the JRC on a document on measuring the impact of science advice.

Outcomes

  • In October 2018, the report from the 2016 workshop on “Publishing models, assessment, and open science” was published, detailing the results of the workshop, which include a framework of questions and a set of concrete recommendations.
  • In May 2018, group members Laura Fierce (USA) and Koen Vermeir (France) presented the results of a survey on the “Study on Reproducibility and Replicability in Science and Engineering”  in Washington, DC. A report “Young Scientist Perspectives on Replicability and Reproducibility in Science and Engineering” is currently under review.
  • Participation in “Policy impact of knowledge and knowledge organisations: From understanding impact towards measuring it”, June 2017 (Koen Vermeir)
  • On a request by ALLEA (ALL European Academies) we gave expert feedback on “The European Code of Conduct for Research Integrity – Revised Edition”, published in 2017. For the factsheet by the European Commission on this document, see: Factsheet on the new European Code of Conduct for Research Integrity. (Martin Dominik, Robert Lepenies, Koen Vermeir)
  • In 2016, the Working Group submitted a well-received expert opinion for the Open Science Policy Platform of the European Commission report on “Next-generation metrics: Responsible metrics and evaluation for open science”, published in 2017.
  • Participation in “Open access and research evaluation: towards a new ecosystem”, Toulouse, October 2016 (Koen Vermeir)
  • In May 2016, group members organised a workshop on “Publishing models, assessment, and open science” in Eindhoven, The Netherlands, as a side event to the GYA Annual General Meeting. A brief workshop report was published in October 2018.
  • In 2015, the group was invited to present the perspective of young scientists around the world on “Rewards, careers, and integrity” in a panel discussion “Research assessment and quality in science: Perspectives from international science and policy organisations” convened by ICSU as part of the “4th World Conference on Research Integrity”, held in Rio de Janeiro, Brazil, from 31 May to 3 June 2015. As a result of a critical analysis by the WG, concrete recommendations for implementation both in the short and long term were brought forward, embracing the link between integrity, transparency, openness, and reproducibility, while warning about pressures that young scientists are facing, and pointing to the effect of reward systems on brain drain.
  • Participation in “The Future of Scholarly Scientific Communication”, The Royal Society, London, Apr/May 2015 (Martin Dominik)
  • Participation in “Beyond Bibliometrics – Identifying the Best“, 8th Humboldt Forum, Berlin, Nov 2014 (Catherine Beaudry)
  • “What is the role of a scientist and what environment is required to fulfill it?”, session organised as part of the careers programme at the 2014 Euroscience Open Forum (ESOF) in Copenhagen, Denmark (Martin Dominik, Catherine Beaudry). As young scientists face a multitude of challenges in their research environments, there is increasing pressure to perform, to conform. Sometimes, diversifying beyond traditional niches is the only way forward. Early-career scientists possess an immense drive to engage, but this potential is not always met by opportunities, and they feel that their wings are clipped. Looking towards solutions for overcoming this dilemma, the session engaged the participants in an interactive debate on the role of a scientist and on the environment required to fulfill it. The dialogue between the two GYA members, Liudvika Leisyte (TU Dortmund) and Alaa Ibrahim (American University in Cairo), as well as the participants of the session touched on issues regarding the size of research grants, mobility, collaboration, gender issues, promotion criteria, peer review, to name a few.
  • “Perceptions of Research Excellence in Thailand and Japan”, STI Policy Review 4, 113, 2013 (Orakanoke Phanraksa, Mitsunobu Kano, Shoji Komai, Wibool Piyawattanametha, Martin Dominik, Rob Jenkins). This publication is the result of the Working Group’s first project, started in 2012. We conducted a survey on the perceptions of research excellence at one leading research institute in Thailand. The findings were used as a basis for improvements in the institutional policy of that institute as well as to further compliment the GloSYS precursor study. In the publication, we assessed the research environment, the promotion of opportunities, and the structures that ensue from assessment criteria.
  • Editorial in “Future of Medical Education Journal” on “Science, Education, and the World’s Future” by Bruce Alberts (FMEJ 2, 2, 2012), soliciting views on “How could young scientists unleash curiosity and creativity to support excellence in science education?“ (Reza Afshari, Marc Creus, Rob Jenkins)