In this post Prof. Henk Moed outlines why the term evaluative framework refers to a specification of the qualitative principles and considerations that provide guidance to a concrete assessment process.
In her latest blog post: There’s no such thing as a bad metric, Lizzie Gadd calls on those who feel committed to the responsible use of metrics to “get off the ‘metric-shaming’ bandwagons, deepen our understanding and press on with the hard work of responsible research evaluation”. I fully agree with her line of reasoning, and would like to bring in a few theoretical distinctions, clarified with typical practical examples, that could hopefully further illuminate the path towards responsible use of metrics in research assessment.
In my new book Applied Evaluative Informetrics (How can we use research performance indicators in an informed and responsible manner?), I defend the position that bibliometrics or informetrics itself does not evaluate, and that in any assessment process an evaluative framework is needed.
The notion of an evaluative framework can be further developed at two distinct analytical levels. The first is a scientific-scholarly foundation of an assessment approach and the informetric tools employed therein. For instance, as far as citation analysis is concerned, such a foundation can be said to be embodied in a ‘citation theory’. To develop such a framework, the book proposes to study the various approaches to the assessment of ‘performance’ or ‘quality’ in other research disciplines, namely business studies measuring business performance, educational research assessing both student and teacher performance, psychological research measuring human performance, and even technical domains assessing technological performance. A core question would be: what can practitioners in the domain of research performance assessment learn from the debates and solutions explored in these other fields?
At the second analytical level, the term evaluative framework refers to a specification of the qualitative principles and considerations that provide guidance to a concrete assessment process. It is on this level that the current blog – and the above mentioned book – puts its focus. A core element in an evaluative framework is the specification of a performance criterion, in a set of propositions on what constitutes research quality or performance. Such values cannot be grounded in bibliometric/informetric research. But from such propositions follow the indicators that should be used, and, in a next logical step, the data sources from which these are to be calculated.
In an assessment process aiming to select from a set of early career scientists the best candidate for a tenured position, for me the important criteria would be: integrity, impartiality; creativity; open mindedness; and capability to reason at distinct analytical levels.
An author who claims that informetrics itself does not evaluate, and that actual assessments should be guided by an essentially extra-informetric evaluative framework, should be cautious when expressing their view on what such a framework should look like, because the danger exists that this would direct the attention too strongly towards their personal views rather than to the claim of the need of such a framework as such, and perhaps even give rise to confounding a principle with one particular realization of it.
In order to stimulate the debate on evaluative frameworks in research assessment, this blog post gives three typical examples of possible elements in an evaluative framework, relating to three distinct application contexts. The choice of indicators – as well as the underlying performance criterion – depend strongly upon the context: what is the unit of assessment; which quality dimension is to be assessed; what is the objective of the process? And what are relevant, general or ‘systemic’ characteristics of the units of assessment?
Example 1. In an assessment process aiming to select from a set of early career scientists the best candidate for a tenured position, for me the important criteria would be: integrity, impartiality; creativity; open mindedness; and capability to reason at distinct analytical levels. These criteria form core elements of an evaluative framework to be used in this assessment. None of these can be assessed with bibliometric indicators, but require in-depth interviews, possibly informed by interview techniques. Evidently, when assessing professional competence, the ability to write and orally present would be important factors too. But citation-based indicators such as H index and journal impact factors are of little use to assess such aspects, because they were not developed as tools to assess early career scientists. Making a solid contribution to a paper in a good, specialist journal (to be established in an interview) would be more significant than a co-authorship in a multi-team paper published in a high impact factor outlet.
Example 2. A national research assessment exercise of a large number of research groups in a particular science field could focus on the bottom rather than the top of the performance distribution, and identify activities in groups or subfields below a certain minimum level. Specification of subject field dependent, minimum levels would establish an evaluative framework. Peer review and bibliometrics could be combined by providing in an initial phase a core peer review committee with a bibliometric study presenting a condensed overview of all groups, and by inviting them to combine this information with their own impressions to select additional committee members, who are experts in subfields about which the bibliometric study raises questions. In a later phase, given the committee’s need to focus its attention for practical reasons, the outcomes of the study could be used to select groups to be interviewed in on-site visits.
Example 3. In some countries, science policies at a national level aim to stimulate domestic researchers to integrate in international networks, and expose their work to critical judgements of an international peer group, by submitting papers to international, peer reviewed journals applying high quality standards. This policy objective sets the evaluative framework, and indicates the evaluative criteria. Although such an objective would not make much sense in the UK, and although several colleagues in the field disagree, the author of this post finds the use of bibliometric/informetric indicators to operationalize the evaluative criteria mentioned in the statement above defensible in principle. This is even true for journal impact factors, provided that they are used to define a minimum level, and take into account a series of pitfalls and technical problems, that could be solved in new, better journal citation metrics.
I hope these examples clarify the notion of evaluative framework, and that they will not be nominated for a “bad-evaluative-framework-Prize”.
Visiting professor, Sapienza Univ Rome; ex CWTS Leiden Univ, ex Elsevier
Unless it states otherwise, the content of the Bibliomagician is licensed under a Creative Commons Attribution 4.0 International License.