Metrics in latest REF documents

In July the UK’s four higher education funding bodies published the draft submission guidance and panel criteria for the Research Excellence Framework (REF) 2021, for consultation before finalising them. Each makes mention of the use of bibliometrics, so I’d recommend reading them to all those involved in the intersection of REF and bibliometrics. As well as these two consultations, there is also a guidance document on environmental indicators.

  1. The Guidance for institutions on environment indicators document gives example indicators in Annex B to include in the REF5a templates – including commitments to responsible metrics, institutional policies on metrics – based on the work of the Forum for Responsible Research Metrics. The REF5a template covers the wider research environment (at the institutional level, rather than per unit of assessment), which will include the open research approach at a given university.
  2. The Consultation on the draft panel criteria and working methods talks about where citation data will be used (remember this is still only a consultation), and there is a range of approaches proposed for different subject areas:
    1. All sub-panels in Main Panel A will use citation data (¶265)
    2. Sub-panels 7, 8, 9 and 11 in Main Panel B “will receive citation data, where available, and may make use of the data as part of the indication of academic significance to inform their assessment of output quality” (¶266). Sub-panels 10 and 12 will not use citation data because they “cannot be used to provide sufficient added value to inform the assessment of output quality” (¶267).
    3. In Main Panel C “Sub-panel 16 (Economics and Econometrics) will receive citation data, where available, and will make use of the data supplied by the REF team where it is considered appropriate as an additional piece of supplementary evidence to support the initial assessment of outputs, not as a determining factor. Sub-panel 16 will take account of the well-known limitations of citations, including equality, diversity and inclusion issues.” (¶268) Other sub-panels will not make use of citation data.
    4. In Main Panel D  they’re not going in for that sort of thing at all. (¶270)
  3. The Draft guidance on Submissions says the REF team will “procure a single source of citation data that provides a good level of coverage across all UOAs in which the sub-panels will make use of such data.” (¶283a) – it will be interesting to see which supplier is chosen. The REF team or its contractor will match outputs to citation counts. Institutions will verify the matches between outputs and citation counts on the submission system, and see the citation counts recorded in the chosen source.

I was struck by a sentence in the submissions guidance document (¶286):

The funding bodies do not sanction or recommend that HEIs rely on citation information to inform the selection of outputs for inclusion in their submissions.

The context of this statement is that citation information will not supplant “expert review” as the primary means of assessment in the REF. I think there is a role for citation data to augment peer review at the institutional level – for instance, in helping to choose between two outputs which appear to be of equal quality – but this would have to take account of the equality implications of using citation data. As in the previous REF exercise, “journal impact factors and other journal rankings” are not to be used in REF assessments (draft panel criteria ¶264). Will people believe it this time round?


Stephen_GraceSteve is the Scholarly Communications and Repository Manager at London South Bank University. He oversees the university’s approach to open access and to research data management, and provides bibliometric support for measuring and evaluating research performance.



Creative Commons LicenceUnless it states other wise, the content of the 
Bibliomagician is licensed under 
a Creative Commons Attribution 4.0 International License.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.