Part one of a three-part blog series summarizing the discussion of the ‘Implementing responsible research assessment’ panel, at the LIS-Bibliometrics 10th anniversary event: The Future of Research Evaluation in March 2020. Some of the questions put to the panel and their responses have been collected and grouped by theme. The panel comprised of Sarah Slowe (University of Kent), Steven Vidovic (University of Southampton) and Karen Desborough (Cardiff University) and this post addresses questions related to implementing responsible research assessment practices and principles.
Q: What can you do if you lack support from institutional management but are enthusiastic to start working towards responsible research assessment yourself?
Without support from institutional leaders it will be difficult to begin genuine start towards change, but it’s possible to strengthen your position.
- Talk about the problem- a lot! Find your champions, influencers, get ideas for possible solutions out there. There will be people who support you who have the position to create change, or who have the ear of people able to create change.
- Identify champions within your institution. For example, focus groups allow you to canvass opinion on the topic and provide data to evidence support for idea.
- Demonstrate the numbers – more than 2,000 organisations have signed up to DORA.
- Demonstrate the necessity. Are these leaders aware that increasingly funders are expecting best practice in research assessment? For example, Wellcome’s open access policy 2021 has imposed a conditions of grant awarding for Higher Education Institutions demonstrate a commitment to responsible research assessment.
- Create one-page briefing papers to target relevant institutional committees or groups with your key messages and supporting evidence.
- Responsible research evaluation shares common goals with equality, diversity and inclusion practice – highlighting this helps these mutually supportive causes strengthen one another.
Q: In your experience, when an institution is considering to make a responsible metrics commitment or policy, is there any internal debate ? If so what are the common points of contention?
Some degree of internal debate is inevitable and not necessarily a bad thing when a Higher Education Institution is making policy decisions that can have substantial impact on process and people.
- Try to consider and solicit a range of views as early as possible in the conversation.
- Examine where non-compliant or problematic practices occur and try to break down why and how they occur. Also recognise that in any large institution, oversight and avoidance of every potential problem may not be possible.
- Develop policy and guidance AFTER consultation.
- Build good working relationships with key stake holders across faculties, as well as vital professional services colleagues such as Human Resources, Communications, ICT.
Q: How do you turn a statement into reality?
In our experience, regular communication, transparency and consistent messaging are key, specifically:
- Well-reasoned arguments. A substantive consultation period should help build this.
- Regular meetings and interaction with people. Keep telling the message!
- Update policies and procedures to ensure they are policy-compliant, as well as actively checking documents and websites to prevent out of date policy or language being shared.
- Pilot new approaches and evaluate their impact, once a policy has been established it should not remain static.
Q: How do you move towards responsible metrics in a context of change fatigue and other problems with working conditions?
Panelists all viewed change as inevitable and positive:
- Change can be positive- it introduces opportunities for diversity and innovation and is necessary to keep up with a changing environment.
- Positive change management includes involving those affected by the principles and outcomes in the development of policy and implementation so they feel in control and invested in the change.
- Promoting and supporting responsible research assessment is one way to help your institution move towards a healthier research culture. Reminding colleagues of the role that assessment has to play along side other important values like research career paths, research integrity and working environment might help.
Q: How can we change the culture to move away from terms like “high impact” and “top journal” as a shorthand for quality? How can we ensure that senior academics promote this cultural change?
- Remove this terminology in written documentation, e.g. amend job descriptions for Teaching and Research staff to remove reference to “high impact” journals as well as other terms that could be construed as such, e.g., “leading” academic journals.
- Promote understanding of diversity in opinions and values. “Top quality” to one person means high Impact Factor, to another it means ABS 4*, to another, it means an international journal with a certain type of editorial board, and to someone else, it means high journal production value. Encourage staff at all levels to share their own opinions and values.
Q: What have you found to be the most challenging thing about implementing responsible metrics?
Cultural change is slow and signing DORA or establishing policies is the beginning and not the end.
- This should not be a box ticking exercise, so feeling like you have made a real difference won’t ever be immediate.
- The goal at the beginning of the responsible metric change will not be the same as immediately or years after the project begins. Metrics evolve, people move on, practice changes.
- Making the policy visible across the organisation, especially outside of the immediate group of supporters and influencers is tough.
- Reaching those who are invested in the traditional journal impact hierarchy.
Sarah Slowe is the Head of the Office for Scholarly Communication at the University of Kent. She has a background in research support and provides support for researchers in maximising the dissemination of their work. Sarah has pioneered the responsible metrics culture at the University of Kent, with a focus on equipping researchers to be able to use appropriate metrics about their work, and the work of others. Sarah is enthusiastic about researcher focussed services, passionate about co-produced research and loves a challenge.
Steven Vidovic is the Open Research Development Manager at the University of Southampton. Steven is responsible for managing and informing the University’s open research strategy and also has interests in publishing ethics and research integrity. Steven holds a doctorate in palaeontology and has previously managed Biological, Earth and Environmental Sciences journals for an international publisher. He is currently the chair of DOAJ’s Advisory Board and contributes to their Council, and is a member of RLUK’s OAPP group.
Karen Desborough is the Responsible Research Assessment Officer at Cardiff University. She is responsible for supporting the implementation of the DORA action plan, working under the direction of the Dean for Research Environment and Culture. Karen’s role includes the delivery of training and information sessions around responsible research assessment and monitoring adherence to DORA principles.
Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.
The Bibliomagician 