Goodbye journal metrics, hello openness? Investigating Plan-S readiness.

Plan-S based funder Open Access (OA) policies claim that they are process-agnostic, with Green and Gold OA both meeting their requirements, but what proportion of your University’s current publishing outlets are Plan-S compliant via the Green OA route and how easy might the transition to immediate open access be? Lizzie Gadd reports on an investigation at Loughborough University.

At Loughborough University we encourage academics to follow a ‘readership, rigour and reach’ approach to choosing where to publish. And to help colleagues assess the reach of an outlet, we may suggest the use of field-normalised journal citation metrics as an indicator of its visibility. But, as an institution with an excellent track record in engaging with Open Access, and a newly minted Open Research Position Statement, we know that openness increases visibility too.  We know that highly-cited journals are only highly-cited because academics have historically submitted their best work there and we are keen to encourage colleagues to think more broadly about routes to visibility.   

Of course, we’re also aware that the external environment is changing and soon the UKRI may be adopting a Plan S-based Open Access (OA) policy which requires the researchers they fund to ensure that the work they produce is made available immediately on publication. This could be through a pure Gold OA journal, a hybrid journal that is ‘transitioning’ to pure, or via Green OA.  At Loughborough, like many medium-sized, less wealthy but research intensive institutions, we have historically embraced the Green route to OA.  Indeed, recent work by the Curtin Open Knowledge Institute using Unpaywall discovered that Loughborough University is 4th in the world in terms of the proportion of our outputs that are available as Green OA.  So, to help us not only guide our academics towards a broader interpretation of visibility, but also to assess our readiness for Plan S, we thought we’d take a look at what proportion of the outlets we currently publish in are not only ’highly cited’ in terms of journal citation metrics[1], but ‘highly open’ in terms of having a zero embargo Green OA policy. 

One thing we didn’t check as part of this analysis was whether those journals offering zero embargo Green OA policies also allowed papers to be made available under a CC-BY licence as preferred by Plan S, and as required by the proposed UKRI OA policy.  This is simply because it is so blooming difficult to get hold of this information.  The obvious place to store it is SHERPA/RoMEO and in some cases that’s where you’ll find it, but coverage is currently very patchy. 

“Currently, just over one-third of our most frequently used sources (35%) would be Plan S compliant, assuming their licences were acceptable.”

So, we downloaded from SciVal the top 100 sources[2] published in by each of our Schools (or disciplines where a School is multidisciplinary) between 2016 and 2018.  We then identified the ten sources in which our authors published most frequently. In some cases, due to differing disciplinary approaches to publishing there were fewer than ten sources in which more than one Loughborough output appeared.  In total 146 sources were identified, and these were checked for citedness (whether they appeared in the top 10% of sources by SNIP or SJR[3]) and for openness (involving a SHERPA/RoMEO search for the length of their embargo period or ‘pure’ Gold status)[4].  

In total, we found that 44% (64) of our frequently used sources were in the top 10% citation percentiles by SJR or SNIP. We also found that 30% (44) had a zero embargo green OA policy as listed on SHERPA/RoMEO and a further 5% (seven) were Gold OA journals.  This would mean that, currently, just over one-third of our most frequently used sources (35%) would be Plan S compliant, assuming their licences were acceptable.  

FIGURE 1: Outlet visibility at Loughborough University

So that was kind of interesting. But, of course, whilst we’re transitioning to new measures of journal visibility, academics will ideally want to focus on sources that are both highly cited and highly open/Plan S-compliant.  So what were their options for hitting both of these indicators? Unfortunately, not so great. Only 22 outlets – just 15% of our most-published-in sources – were both highly cited and highly open, with one additional outlet hitting the ‘highly open’ target by virtue of us paying for the privilege (APC-based Gold OA).  

When we shared this with academics their perhaps inevitable next question was, well what highly cited and highly open options do we have across the wider list of 100 sources (i.e. not just those we’re publishing in the most)?  Surely if we widened the net we’d find much greater opportunity to grasp ‘mega-visible’ publishing opportunities? Alas, it was not to be. Having extended the (very time-consuming) exercise to check all their highly cited sources for open access options we found that a much smaller proportion overall, a mere 7%, hit both indicators.  This varied of course from discipline to discipline with the greatest opportunity being afforded to communications (16%) and the lowest to education (0)[5].  And of course, this is before we factor in whether those zero embargo Green OA titles actually allow manuscripts to be posted under CC-BY licences.

Oh dear.

Now I’m aware we have a sample of one university here. And our publication practices may or may not be representative of the wider population.  Indeed, it would be great if others could run this analysis at their institutions to see how widespread this phenomenon is. But whether or not it’s replicated elsewhere, this is the reality for us.  

The low-hanging fruit of course, is to draw attention to those 54 titles (37%) that hit neither visibility indicator.  And by broadening our ‘definition’ of visibility, we can highlight a wider range of titles that can serve this important end.  However, if we were hoping to find a good list of titles in which academics were currently publishing that were both highly cited and highly open, we were pretty disappointed.  On average, each of our disciplines had five sources to choose from that were both highly open and highly cited.  In reality, some had none at all. So what do we do with that?

“Having extended the exercise to check all their highly cited sources for open access options we found that a mere 7%, hit both indicators.“

The truth is that although we use citedness and openness as visibility indicators, they do both indicate different aspects of visibility. Openness speaks of potential reach and, if other open research practices have been engaged with, perhaps increased rigour.  Citedness speaks of actual reach, of journals that have a track record of finding and influencing their target audience and, because they attract so many papers and have built up stringent peer review processes to weed out the poorer ones, they may also claim increased rigour. So, again, what to do? 

I think that all too often we research support folk can hide behind our general principles and our generic advice: “We support openness”. “Consider open access options in your publication choices.”  But if an academic collars you and asks explicitly whether they should choose Journal A that is highly cited and closed, or Journal B that is poorly cited but open, and assuming the readership and rigour of both are comparable, we find ourselves in an extremely tricky spot, caught between conscience and convention.

And that, my friends, is why to meet the demands of Plan S (and the UKRI OA policy) I fear we are going to have to abandon Green OA in favour of pricey “publish & read” or “read & publish” big deals with the publishers of existing highly cited journals. There simply aren’t the zero embargo Green OA deals around for the sources in which we publish the most. And again I iterate, this is before we’ve factored in the potential CC-BY requirement. I think it’s unlikely that publishers, given the choice between making their Green OA policy zero-embargo-with-CC-BY or receiving additional ’gold-for-Gold’, are going to opt for the former.

It might be helpful to policymakers such as the UKRI to understand how widespread this experience is, so if anyone fancies running this analysis on their own HEI, I’ve provided my method below. Similarly, if you felt able to share your data when you’re done, I’d love to hear from you.

 

Huge thanks to Dr Karen Rowlett for her comments on an early draft of this piece. 

[1] As I say in the opening paragraph, there is really no such thing as a highly-cited journal, only journals to which academics submit their best work, that ends up lending its citedness to that journal. However, I use the term ‘highly cited’ as a short cut. Don’t judge me.

[2] SciVal only allows you to extract 100 sources per entity.

[3] Each School and discipline at Loughborough gets to select (or not to select) their own field-normalised journal metric and threshold.

[4] If a source was not listed on SHERPA/RoMEO it was recorded as being non-compliant as we didn’t have the resource to chase down every title individually. We also did not factor in any existing publisher deals that allow Loughborough academics to publish ‘APC-free’. This in reality, the percentage of Plan S-compliant titles might be a bit higher than this.

[5] Excluding politics, history and social work where SciVal’s coverage of our titles is too low to be meaningful.

Method 

(I used SciVal but you could also use a bibliographic database such as Dimensions, Scopus or Web of Science)

• In SciVal – Overview – 2016-18 – Published – By Scopus Source – Export the data one School/department/discipline (hereafter, research unit) at a time into Excel
• For each research unit, highlight the ten outlets in which they publish the most
• Highlight (using number filters) which of those ten titles appear in the top 10% by the journal citation metric of their choice. (We use data provided by SciVal to help us with this. We use 1.5 as the overall threshold for top 10% SNIP and 1.4 for top 10% SJR. You could use discipline-specific thresholds if you preferred.)
• Search for each title on SHERPA/RoMEO and check whether it:
  • Allows self-archiving of the Accepted Version immediately on an Institutional Repository. 
  • Is a pure Gold OA Journal. On SHERPA/RoMEO this is indicated by ‘Listed in DOAJ? Yes’
• You may also wish to note the number of titles not listed on SHERPA/RoMEO.
• Record the total and percentage of outlets that:
  • Appear in the top 10% by journal citation metric
  • Have a zero embargo Green OA policy
  • Are pure Gold OA journals
  • Are both top 10% and zero embargo OR are both top 10% and Gold OA journals.

• You may also do this for your whole institution if you are unable to disaggregate by field.  To do this:
  • Download a list of sources published in by your HEI between 2016-18
  • Extract their journal metrics using the Scopus source list download.
  • Filter on those in the top 10% (use thresholds provided above).
  • For those top 10% sources, check SHERPA/RoMEO for their Green/Gold OA status as above.
  • Record the total and percentage of outputs that:
    • Appear in the top 10% by journal citation metric
    • Have a zero embargo Green OA policy
    • Are pure Gold OA journals
    • Are both top 10% and zero embargo OR are both top 10% and Gold OA journals.

---

Elizabeth Gadd is the Research Policy Manager (Publications) at Loughborough University. She is the chair of the Lis-Bibliometrics Forum and co-Champions the ARMA Research Evaluation Special Interest Group. She also chairs the INORMS International Research Evaluation Working Group.

Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.