“Don’t stop thinking about tomorrow!” – Future Competencies in Bibliometrics

Hannelore Vanhaverbeke, Barbara Lancho Barrantes & Andrew Cox are set to update the “Competency model for bibliometrics work” and after gathering feedback they present their early findings here.

A Time for Review

In view of the fast-paced changes in bibliometrics, both in terms of new indicators, the ‘next-generation metrics,’ and in terms of new technology, such as artificial intelligence and data mining, the question arises as how all of this impacts the future of research evaluation and the competencies bibliometric practitioners need to develop.

In 2017 the Competency model for bibliometric work was developed by Andrew Cox (Sheffield) and Sabrina Petersohn (Wuppertal). This model was the outcome of a study commissioned by the LIS-Bibliometrics forum, sponsored by a research grant from Elsevier Research Intelligence Division. It has since then functioned as a self-assessment tool for bibliometricians, a way to identify (and remedy) training needs in organisations, and an aid to setting up study programmes.

“It has been 4 years since the Competency model was released”

Three levels of involvement with bibliometrics were defined: an entry level, a core level, and a specialist level. Expected knowledge and skills were defined across 4 themes: the scope of applying bibliometric knowledge, the expectations relating to advocacy and training responsibilities, the level of technical knowledge of bibliometrics tools, and the overall professional conduct. The model was presented explicitly as a living document up for regular reviewing. It has been 4 years since the Competency model was released and it was considered to be time for an evaluation of the model’s content.

Gathering Feedback

The perfect occasion presented itself with the scheduled LIS-Bibliometrics conference on The Future of Research Evaluation in Leeds in March 2020, at the 10th anniversary of the LIS Bibliometrics group.

Prior to the meeting we set up an online survey asking whether the competency model was known and used, and how it was perceived by its users, and asking respondents to identify their needs for training and professional development in the near future. An invitation to participate was sent to diverse community lists and a total of 130 responses (16/03/2020) were received.

During the Leeds meeting, both people present and people attending the workshop, “Don’t stop thinking about tomorrow – Future competencies in bibliometrics” were asked online for feedback to a subset of the survey questions and to some additional questions. These related to the gaps in training they experience and to challenges identified within the realm of research evaluation. We received between 60 and 110 answers to each of these questions (not everyone responded to all the questions during the workshop).

Preliminary Findings

While the full results will be presented in a paper yet to be published, three elements are of relevance here:

1)     The uptake of the competency model

Only a third of the respondents to the online survey (35%) had heard of the model, and this mainly by way of the LIS-Bibliometrics mailing list.

Within this group, the model was primarily used for self-assessment (66.7%). Other applications that were repeatedly mentioned, be it to a lesser degree, were its contribution to identifying new skills to be developed in a team (35,6%) and to developing training programmes for staff involved in bibliometrics (28.9%).

“In view of these observations…it seems worthwhile to devise strategies of giving more visibility to the model.”  

Users of the model rated it highly for self-evaluation (4.31/5) and identifying roles and new skills within a team (4.2/5) and for helping organisations to develop training programmes and support their staff in developing the necessary bibliometric skills (4.07/5).

In view of these observations – the limited acquaintance with the model combined with the high scores given by its users – it seems worthwhile to devise strategies of giving more visibility to the model.  

However, noting the slightly lower scores given by the survey participants for the completeness (3.65/5), up-to-dateness (3.7/5), user friendliness (3.42/5) and applicability (3.6/5) of the model, some revision in content and style seem appropriate.

2)     The need for additional training was a common thread

Fifty out of 54 respondents during the workshop stated that they would like further training. This may be partly related to the fact that most people working in bibliometrics have not received specific training for the role, but this may not be the main reason. As said above, technology changes so fast and new methodological insights are generated at a steady pace. Continuous education is therefore the norm.

Among the participants to the online survey and the workshop, the top 5 identified fields for which extra training was deemed necessary were:

  • New applications for data analysis, with associated statistical literacy (132 votes)
  • Data visualization tools (82 votes)
  • Issues of ethics, responsible use of metrics and the communication of these (78 votes)
  • Coding and the use of AI (74 votes)
  • Making use of APIs for more efficient or even automatic data capture (58 votes)

What is striking is the dominance of technical training needs, all in the field of data science, beyond a greater knowledge of bibliometric indicators, content and application-wise. The main issue relating to the latter is the need for more guidance, clearer statements, and practices on the road to a more ethical and more responsible use of the metrics.

As to the format that training would ideally take, the consensus overall was the use of interactive tools with exercises (56 in Leeds, 74 survey) and live webinars (34 in Leeds, 67 survey), followed by recorded tutorial clips (30 in Leeds, 62 survey), and physical meetings, in pre-COVID times, ranked 4th (23 in Leeds, 42 survey).

The clear choice for this training format poses challenges: not many interactive courses with exercises for self-assessment are available. So maybe this is the most concrete action for the bibliometrics community: the development of such a tool, independent from commercial providers. But who is going to be able to invest time and money in this?

3) Major challenges facing the future of research evaluation

In addition to questions around the development of competencies, participants were asked what they saw to be the major challenges facing the future of research evaluation.

The first major challenge that participants to the Leeds workshops identified (the online survey did not ask this question) was the difficulty in tackling irresponsible practices both at senior management level and researcher-level when it comes to using bibliometrics (102 votes). These concerns range from an over-dependency on seemingly easy to interpret metrics (such as in league tables or most commonly used indicators) through to the level of data accuracy, lack of transparency and reproducibility, gender- or career-stage biases, and the disregard of discipline specific scientific practices.

“The first major challenge identified…was the difficulty in tackling irresponsible practices both at senior management level and researcher level.”

The second major challenge facing bibliometric practitioners was the ever-expanding monopoly of publishers in the area of bibliometrics and the resulting commercial ownership of indicators for research evaluation (70 votes). Bibliometrics and research evaluation is big business – leading to another issue identified: a proliferation of new metrics, platforms and tools to assess well…just about anything. Since most of these elements are owned by commercial institutions, access to these is mainly granted through subscriptions. Subscriptions that not every HEI can afford.

“The second major challenge…was the ever-expanding monopoly of publishers…and the resulting commercial ownership of indicators.”

So, where does this take us? What can we conclude from these insights?

The Competency model is up for both a cosmetic update and a major overview of the competencies required by the modern bibliometric practitioner. Users of the model rated it highly, but the user-friendliness did not score well. A more interactive website could do a lot to attract people to explore its contents, as well as adding links to existing education online sources can be a first step. Content-wise new developments such as coding, AI, can be easily added to the framework. If this is done, a promotion to different fora and mailing groups is needed.

“Two trends are very clear: more training in what is now called data science, and more practical guidance on the ethical and responsible use of metrics.”

Focusing on the skills bibliometricians are in need of today, two trends are very clear: more training in what is now called data science, and more practical guidance on the ethical and responsible use of metrics. Can we interpret this to signal that bibliometricians have enough indicators at their disposal, and that the need is now for more efficient data analysis and a deeper understanding of the (statistical) meaning of these data? With a good dash of input from the ethics community? Do we all need to become ethically aware data scientists? It seems that yes, bibliometrics is to become more of an application of data science than it has been before. Today’s more specialist bibliometricians may be expected to have a basic knowledge of programs such as R or Python, of data visualization techniques, of what AI can do, how APIs can be used, all of this combined with a good statistical insight.

“What is clear is that the LIS-Bibliometrics community has still got a significant part to play in supporting, training and skilling bibliometric practitioners.”

For many of us who have been working in the field for a long time, this possibly feels scary. It should not be for those of us who work in a team where one of the colleagues is experienced in these applications. For others working in relative isolation, the issue is more problematic. Maybe new strategic alliances with IT, or even researchers working in the field of data science can provide a solution? Tutorials with exercises can also remedy this gap but (as far as we know) there aren’t many of us who have the time to develop these. However, one could state that, as bibliometricians, it is our duty to ‘take care of one another’ to keep the community a viable one. Efforts in getting everyone on board could easily be spread – sharing news of online courses, organising webinars ourselves, collaborating beyond our own institution. LIS-Bibliometrics can investigate how such a concerted effort could take shape.

What is clear is that the LIS-Bibliometrics community has still got a significant part to play in supporting, training and skilling bibliometric practitioners to be the best that they can be both technically, and in terms of negotiating some of the challenges around the irresponsible use of metrics and in perhaps supporting the development of new, community-based products. It is an exciting time to be a bibliometrician!

Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.

Hannelore Vanhaverbeke (PhD Archaeology) is head of the Data Analysis Unit at KU Leuven’s Research Office (Belgium). This unit manages and monitors research related data (publications and citations, PhDs, research expenditures, projects) and provides support for funder compliancy in terms of data management and Open Access. As a science policy advisor on Open Science, scholarly communication and research assessment, she is active in diverse platforms: she co-founded the Flemish Interuniversitary WG on Research Data Management (RDM) and Open Science (OS), is a member of the Flemish Open Science Board WG on RDM & OS, and of LERU’s Information and Open Access Policy Group. AT KU Leuven she coordinates the Open Science WG on Rewards & Incentives. Above all, she strives to infuse vision with a strong dose of pragmatism and a dash of humour.

http://orcid.org/0000-0002-8748-8808

Barbara S. Lancho Barrantes is a Bibliometrician leading the Raising Research Visibility and Bibliometrics service at the University of Leeds (United Kingdom). She has a PhD in Bibliometrics developed within the SCImago research group. Barbara is a competencies development officer in LIS Bibliometrics committee and a member of the Metrics toolkit editorial board. 

https://orcid.org/0000-0001-9994-8886

Andrew M Cox is a senior lecturer at the Information School, University of Sheffield (United Kingdom). 

orcid.org/0000-0002-2587-245X

Unless it states other wise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.