Part three of a three-part blog summary of the ‘Implementing responsible research assessment’ panel at the LIS-Bibliometrics 10th anniversary event: The Future of Research Evaluation in March 2020. Some of the questions put to the panel and their responses have been collected and grouped by theme. The panel comprised of Sarah Slowe (University of Kent), Steven Vidovic (University of Southampton) and Karen Desborough (Cardiff University). This post answers questions the panel received on communicating and monitoring compliance after establishing a responsible metrics policy or signing DORA.
Q: Who lead the external communication of adoption of DORA?
- Each institution had slightly different approaches (read post 2 of the series) and different senior sponsors at the initial stages of implementation, but all agreed that it was necessary to have as much buy-in as possible from across the entire institution and as may lines of communication as possible with university leadership.
- Senior academic champions can make a big difference for ‘buy-in’.
- All methods of engagement should be explored as far as possible including Human Resources, Communications and Marketing to help spread the message and ensure the right language is being used in university communications.
Q: How are local policies governed and how do whistle-blowing processes work? Who receives the complaints and how they are investigated?
- Cardiff University and University of Kent opened dedicated email inboxes for raising concerns over the use of responsible research assessment. If necessary, Cardiff has policy is in place to escalate to the Open Research Integrity and Ethics Committee, which has oversight of DORA.
- Usual processes for local reporting and escalation are expected to take place for the misuse of metrics, as with any other form of misconduct. However, none of us have ever experienced this level of misuse of metrics and find educational and early intervention more effective.
- University of Southampton had pre-existing whistle-blowing and research misconduct procedures and policies, which meant that the new Responsible Research Metrics Policy could simply point to those. The Library is responsible for maintaining expertise for matters of troubleshooting.
Q: A criticism of DORA is that it has too heavy a focus on Journal Impact Factor and can neglect problems of appropriate use of other indicators. How can a university deal with this?
- We agree that DORA is very focused on journal-based metrics being used inappropriately to assess individuals, but this reflects the severity of this issue in the academic evaluation system. We support DORA’s explicit wording and terminology against Journal Impact Factors.
- The declaration also lays out positive prescriptions for reforming research assessment in an inclusive and transparent way.
- Responsible use of metrics is not a ‘DORA or nothing’ approach – there are many paths and the most appropriate one will differ depending on the institution. DORA has key strengths and the Leiden Manifesto, the Metrics Tide and institutional approaches are all equally strong approaches for a diversity of institutions to choose from and adapt.
- University of Kent chose to adopt the Leiden Manifesto as the preference of their academic community.
- University of Southampton developed a policy and methodology to appropriately assess research and individuals producing it in all situations. This policy contains clauses that prohibit the use of journal-based metrics in the assessment of an individual or an individual piece of research because of the prevalent misuse of Impact Factors.
Q: How can institutions can go beyond DORA to be more inclusive of Arts, Humanities and Social Sciences (AHSS)? Both DORA and many metrics are very Science, Technology, Engineering and Maths (STEM) focused, but end up being applied to AHSS.
- It is a fair criticism and one which we believe DORA recognises – we understand DORA is currently reaching out to humanities scholars and has a vision is to work across all scholarly disciplines.
- The concepts in DORA are not explicitly exclusive of AHSS concepts, they’re just not entirely representative because AHSS scholars face a different academic environment than STEM scholars.
- In a local institutional context, consult with representatives of AHSS of all seniority levels to get a diversity of views and challenge their thinking about how they are using different types of metrics. We feel that the Leiden manifesto can reflect AHSS concerns well.
Sarah Slowe is the Head of the Office for Scholarly Communication at the University of Kent. She has a background in research support and provides support for researchers in maximising the dissemination of their work. Sarah has pioneered the responsible metrics culture at the University of Kent, with a focus on equipping researchers to be able to use appropriate metrics about their work, and the work of others. Sarah is enthusiastic about researcher focussed services, passionate about co-produced research and loves a challenge.
Steven Vidovic is the Open Research Development Manager at the University of Southampton. Steven is responsible for managing and informing the University’s open research strategy and also has interests in publishing ethics and research integrity. Steven holds a doctorate in palaeontology and has previously managed Biological, Earth and Environmental Sciences journals for an international publisher. He is currently the chair of DOAJ’s Advisory Board and contributes to their Council, and is a member of RLUK’s OAPP group.
Karen Desborough is the Responsible Research Assessment Officer at Cardiff University. She is responsible for supporting the implementation of the DORA action plan, working under the direction of the Dean for Research Environment and Culture. Karen’s role includes the delivery of training and information sessions around responsible research assessment and monitoring adherence to DORA principles.
Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.
The Bibliomagician 