Katie Evans looks at the wider context of responsible metrics and asks what the ingredients of a culture of responsibility might be.
I talk to academics about bibliometrics on an almost daily basis. One conversation that stands out for me started with the academic asking to meet me because they “disagreed with metrics”. To my surprise, in the course of the conversation it emerged that this academic was happily and sensibly using a range of bibliometrics in the course of their work. What they were concerned about was not metrics per se, but trends they observed in academic culture worldwide: ‘publish or perish’ demands pressurising early career researchers and unduly influencing which projects they took on; heavy handed performance management poisoning morale; excessive weight placed on league tables etc. All issues that can arise in what The Metric Tide refer to as “cultures of counting” (Chp 7).
The Forum for Responsible Research Metrics’ recent The Turning Tide event felt like a similar conversation on a larger scale. For me, the highlight of the day was the panel of early career researchers who spoke eloquently about wanting to reclaim the wonder of research from the tyranny of pressure to focus single-mindedly on publishing in high impacts journals; wanting to be recognised as individuals with talents and potential, not reduced to a scorecard of numbers. The criticism was of organisational/academic culture rather than metrics per se. Clarity and competency based evaluation criteria (that might include metrics) were valued as a way to challenge prejudice. And I didn’t hear any objections to Professor Adam Tickell’s statement that we need evaluations such as the REF to make the case to government for continued investment in research.
So what does make a good master? Here I find myself reaching for intrinsic values like fairness, integrity and adding to knowledge and understanding.
My conclusion is that metrics make good servants but bad masters. In this, I’m echoing the Leiden Manifesto: “The problem is that evaluation is now led by the data rather than by judgement … We risk damaging the system with the very tools designed to improve it” (p429). I would say the same applies to peer review and expert judgement of research quality – good servant, bad master. Think of the issues of unconscious bias and outright prejudice, or of the risk of stifling developments that question theories cherished by the current experts.
So what does make a good master? Here I find myself reaching for intrinsic values like fairness, integrity and adding to knowledge and understanding. Perspectives that recognise the humanity of everyone involved in research and value their contribution, not treating anyone instrumentally as a mere cog in a machine.
I’m proud of the University of Bath’s statement of principles of research assessment and management. It gives me solid ground to stand on to reassure researchers gripped by nebulous fears of being assessed by numbers: they and their work won’t be reduced to a flat scorecard, we are committed to higher standards than that. But I don’t fool myself that a statement of principles can by itself sustain a healthy organisational culture.
Responsible metrics statements and responsible bibliometric practitioners are good ingredients to go into the mix, but you wouldn’t make a cake from just sugar.
I’m in complete agreement with Lizzie Gadd’s call for responsible bibliometric practitioners. But if our concerns and the research community’s concerns are primarily about academic and organisational culture, then we need to be talking about culture as well as metrics. How do we create “cultures of responsibility” (The Metric Tide, 7.7)?
Responsible metrics statements and responsible bibliometric practitioners are good ingredients to go into the mix, but you wouldn’t make a cake from just sugar. What else do we need to balance out the risk of a (potentially sickening) dominance of metrics?
Answering this question will be a collaborative effort. We need responsible policymakers, responsible university management and responsible research leaders as well as responsible bibliometric practitioners using metrics responsibly.
References
- Hicks, D., Wouters, P., de Rijcke, S., & Rafols, I., 2015. The Leiden Manifesto for research metrics. Nature. 520 (7548) 429-431 DOI: 10.1038/520429a
- Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363
Katie Evans is the Research Analytics Librarian at the University of Bath, responsible for supporting the use of research publishing, citation and collaboration data across the University. Previously, Katie worked on open access at the University of Bath.
Unless it states otherwise, the content of The Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.
The Bibliomagician 