Lizzie Gadd argues that good research evaluation starts with good questions.
The 2019 Ig Nobel Prize winners were announced in September. Among my favourites was some research into the pleasurability of scratching an itch. Contrary to what the name suggests, winning an Ig Nobel prize is not an indictment of your research design or methods. It’s an indictment of your research question. Is the question a good one? Is it an important one? Or are you even starting with a research question at all?
It strikes me that in research evaluation we’re also in danger of this. Often, it’s not our metrics that are at fault, or even our methods, it’s our questions.
The INORMS Research Evaluation Working Group recently developed a process called SCOPE by which research evaluation can be done responsibly. The S of which stood for ‘START with what you value’. In other words, start with the right questions, the things you want to know, and work forwards. Don’t start with what others value, or with your dataset, and work backwards.
So, a discussion list member recently reported that they’d been asked to come up with a single research indicator for their university’s KPIs. Just the one. ‘Research indicator’. What does that mean? What do they want to indicate about research? What are they trying to achieve? There are a whole lot of pre-questions that need answering before we can start to answer this one.
No-one questioned whether arts & humanities colleagues actually valued citations enough to want to be measured by them.
Again on another discussion list, a colleague was seeking a bibliographic data source which provided better coverage of arts and humanities content for bibliometric analysis. Lots of helpful folks pitched in with responses. But no-one questioned whether arts & humanities colleagues actually valued citations enough to want to be measured by them, or whether there were better ways of assessing the quality and visibility of their outputs. Wrong question.
But it’s not just asking the wrong question that we’re prone to, it’s not starting with a question at all and retrofitting one once you have your data. In science this has become known as ‘HARKing’ – Hypothesising After the Results are Known. And I’ve seen two cases recently where it feels like this is happening.
So, Elsevier’s newly formed International Centre for the Study of Research recently produced a report demonstrating the increase in fractional authorships resulting from increased collaboration. Fine. Except it was advertised as a study purporting to answer the question “Are authors collaborating more in response to the pressure to publish?”. Now this is a good question, but you can’t answer this question with the data they have. If you want the answer to THIS question, you’d have to ask authors what their motivations for collaborating were. The study doesn’t do that. Wrong question.
In the interests of balance, a collaboration between Digital Science and CWTS Leiden recently produced a research landscape mapping tool. It’s a very interesting visualisation of the research publications resulting from the big funding agencies. Great. However, it was promoted as a tool to “support research funders in setting priorities”. Now, knowing the brilliance of CWTS Leiden, I struggle to believe that they started with the question ‘How can we better support funders to make funding decisions?”, and ended up with a tool that showed what publications had resulted from historical funding decisions. And the thought that this data should be used in any significant way to support funding decisions concerns me. What seems more likely to have happened is that a fabulous dataset and a fabulous visualisation tool got together and had a fabulous research mapping tool baby, and then, to justify the accident of its birth, cursed it with a grand title it could never quite live up to.
Asking the right question is important. And we need to articulate our question before we decide how best to answer it. And when we answer it, we need to offer our findings in light of the question we asked, not one we think might give the better headline.
Ultimately we need to look beyond the questions we are asking to the systemic effects of asking them. Let’s take the h-index as an example. My h-index is currently 13. And if my h-index of 13 is the answer, what is the question exactly?
There’s a bigger question as to whether rewarding only publication-producing individuals is good for scholarship? Good for our universities? Good for humanity?
So, technically, I guess the question is, “How many publications do you have with at least that number of citations?” But the implied question is “How prolific are you at producing well-cited publications?” However, you can’t answer that question without knowing my career stage and discipline and how many career breaks I’ve had and what role I played on those publications and how many co-authors they’ve had. So the h-index alone doesn’t provide an answer. But even if it did, the systemic effects of asking this question are significant. Do we want to employ or promote individuals who are only prolific at producing well-cited publications? The answer might be yes, because that’s how we as universities are measured. But then there’s a bigger question as to whether rewarding only publication-producing individuals is good for scholarship? Good for our universities? Good for humanity? Is this actually fulfilling our mission as universities? Even if it’s making our universities look good in the eyes of the rankings or some other well-meaning but misguided party.
Aaron Swartz once said, “what is the most important thing you could be working on right now? And if you’re not working on that, why aren’t you?”. It seems to me that the biggest problems in the research evaluation space could be solved not by better methods and metrics, but by better questions. Are they important? Are they honest? Are they mission-driven? Once we have our questions right – once we have put our values back into our evaluations – we will be well on the road to more responsible research assessment.
Elizabeth Gadd is the Research Policy Manager (Publications) at Loughborough University. She is the chair of the Lis-Bibliometrics Forum and co-Champions the ARMA Research Evaluation Special Interest Group. She also chairs the INORMS International Research Evaluation Working Group.
Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.