Navegación – Mapa del sitio

InicioNuméros44-2Actualité de la rechercheDébats. L’évaluation des publicat...Evaluation of research publicatio...

Actualité de la recherche
Débats. L’évaluation des publications scientifiques en débat

Evaluation of research publications and its impact on research: the case of Great Britain

Geoffrey Crossick
p. 287-291

Texto completo

  • 1  G. Crossick is Distinguished Professor of the Humanities in the School of Advanced Study, and was (...)

1The evaluation of research publications has been discussed in Britain for a number of years, but unlike some other European countries, this has not been in order to establish a formal system for evaluating or ranking journals or book publishers.1 The discussion has taken place in the context of the national Research Assessment Exercise (RAE) and its successor the Research Excellence Framework (REF), which take place at five- to seven-year intervals, and where individual research publications are evaluated. The place of publication (the journal, the publisher) ostensibly has no part in this process. A further context for the evaluation of research publications is that of career development and promotions for academics, which in the UK is entirely a matter for individual universities with no national framework or requirements. Research judgments related to career development are modelled on the expectations of the RAE/REF, because the national system of research evaluation is the most important basis not only for departmental and university reputations, but also for a university’s research income from the Higher Education Funding Council for England (HEFCE) and its equivalents in other parts of the UK.

2The system of research evaluation in the UK began in 1986 and has evolved in the nearly three decades since then. The primary purpose of the RAE was firstly to raise the quality of research in UK universities, and secondly to provide a formulaic mechanism for the distribution of a significant proportion of government funding for university research. The two objectives were intimately connected. The mere fact of evaluation, even if the results are made public, is not as powerful a force for improvement without other major incentives for strong performance. The financial incentive is an effective one, and universities have responded as intended, with each seeking to maximise its performance in successive RAEs.

3Under REF 2014, each university decides which departments and which members of staff to submit for evaluation. There must be four publications (or other research outputs) for each individual submitted. Rigorous guidelines determine when fewer than four outputs are required, mostly concerned with the researcher being at an early career stage, or having had interruptions due to maternity, care responsibilities, working in industry and so on. In addition to the research outputs, which will count for 65 per cent of its overall evaluation, each department will also be evaluated for its «research environment» (15 per cent) and—a new element for 2014—the wider impact of its research (20 per cent).

  • 2 For full details of the methods used by the main panels and sub-panels, see HEFCE, REF 2014. Panel (...)

4HEFCE (which manages the REF on behalf of the higher education funding bodies in the other three parts of the UK) has established 36 disciplinary «Sub-Panels», overseen by four «Main Panels», and it is the sub-panels that will carry out the evaluation of the research outputs.2 The system rests on peer review («expert review» in the documentation). Each sub-panel comprises about 20 members, mostly academics but with some members drawn from the research-user community. Although international academics are occasionally members of sub-panels, it is on the four main panels, which oversee the work of the sub-panels to seek consistency of process and standards, that one finds a strong international presence. Some 27 per cent of those on main panels are drawn from outside the UK—from North America and other European countries including the Netherlands, Germany, Belgium, Italy and Denmark.

5The members of the sub-panels will spend several months evaluating the outputs submitted, along with the submissions on research environment and research impact. They will meet to share their findings and to finalise their grade profile for each department. HEFCE established the membership of the main panels and sub-panels after consultation with national academies, subject associations, universities, industrial and professional bodies and others. It would be surprising if all were entirely satisfied by the outcome, not least in a context where some balance has to be struck within each sub-panel between subject specialisms within disciplines, the different parts of the UK, gender and so on. Nonetheless, it has to be said that the expert review sub-panels by and large command respect. At least, that is, until their judgments are known.

6Although the work of the sub-panels is confidential, considerable attention has been paid to the transparency of the process in establishing their membership, to communicating the basis and criteria on which they will work, and to the processes required when universities make their decision about whom to submit. Although this has not prevented controversy at local and national level—not surprisingly as they involve individuals and their careers—the concern for due process, transparency and equality of opportunity reflect a mixture of genuine principle, the search for legitimacy and a fear of litigation.

7It is in this context that the evaluation of research publications takes place. The term used by the RAE and now the REF is not research publications but «research outputs». This enables a much wider range of products of research to be submitted for evaluation. Not just journal articles, books and peer-reviewed conference proceedings, but also patents and designs, reports for government or industry, and practice-based outputs in the arts where these are based on research (musical compositions, exhibitions, performances and so on). Although it is difficult to evaluate outputs that often cannot be experienced by the evaluators, procedures have been established to enable that to be done.

8How are the research publications, which constitute the great majority of outputs submitted even in creative and performing arts disciplines, themselves evaluated? It is the works themselves that are evaluated, with no formal account taken of the standing of the journal or publishing house where they appeared. The intention is that each publication be read sufficiently for a sound judgment to be made, and many sub-panels, especially in the humanities and social sciences, give the impression of expecting to read in full most of the work submitted. The reality may be different, with sub-panel members using a variety of unvoiced techniques (abstracts, tables and graphs, and an informal feeling from the location of publication before reading parts of a publication) to come to a conclusion about the quality of the output. A significant proportion of the publications will nonetheless be read in their entirety, especially in the arts, humanities and social sciences.

  • 3 Development of Bibliometric Indicators of Research Quality. A Report to HEFCE by the Centre for Sci (...)

9In the early days of planning the new REF there was talk of using quantitative bibliometric indicators to make the workload lighter and the outcomes more objective. This assumes that quantitative data is necessarily more objective and that quality can be divorced from personal judgment, ignoring Aristotle’s injunction to seek precision in an inquiry only in accordance with the limits set by the character of the inquiry itself. It became clear, however, that this search for the bibliometric Holy Grail would not succeed. First, because neither of the most common techniques—citations and journal impact factors—made much sense for the arts, humanities and most of the social sciences, that is to say for the majority of the research community. It also became evident that these approaches were doubtful even in many disciplines in the natural, physical and engineering sciences. This was made clear by work commissioned by HEFCE from the Centre for Science and Technology Studies at Leiden University.3

10The new REF thus stepped back from an approach driven by quantitative data. Individual sub-panels will be permitted to use bibliometric data to inform their peer-review judgment of outputs, but not to replace them. The discussion was always about citations and journal impact factors, and there was no significant attempt to establish a classification of journals, let alone any formal hierarchy. The result has been a considerable degree of caution about the pan-European initiatives for journal classification. There has been even less interest in classifying types of book or a formal hierarchy of publishers. Individual members of sub-panels may well have hierarchies of journals and publishers in their mind when forming their expert review judgments, but these will be merely informal and sub-panels will use their formally-identified criteria. A typically Anglo-Saxon fudge? Perhaps, but the alternatives would have been deeply divisive and probably spurious in their claims to accuracy or authority.

11Responsibility is, in effect, passed to each university to ensure that the outputs it submits look as strong as possible, an approach that can lead to some conservatism about the location of publications. They will have prepared carefully, often using advisers from other universities, and these will doubtless have given particular weight to the location of publications as internal judgments may not have been based on extensive reading of the outputs.

  • 4  HEFCE, Ref 2014, p. 25.

12Does this mean that sub-panels will take no account of bibliometric information? They were given the option of using it to supplement expert review judgments, and the 36 sub-panels have each indicated their approach. All 36 have definitively rejected the use of journal impact factors. As far as citations are concerned, the natural, physical and life science panels will make more use of them than those in the arts, humanities and social sciences. In the words of Panel A (medicine, health, psychology, biology, agriculture) «this will only be one element to inform peer-review judgments about the quality of the output, and will not be used as a primary tool in the assessment.»4 All Panel A’s sub-panels will use citation data, drawn from the Scopus citation database, but state that they will not use any other sources of bibliometric data. In Panel B (chemistry, physics, maths, computing and engineering) there is more variation, with four sub-panels using citation data and five rejecting it. Once we get to Panel C (social sciences) and Panel D (arts and humanities), citation data is rejected by all 21 sub-panels except for Economics.

13If the REF (and, previously, RAE) is the key moment for the evaluation of research outputs, the other moments concern individual careers. At each key career stage—first appointment, promotion to senior lecturer and promotion to professor—every university uses its own procedures and criteria. There are no formal national criteria, let alone any bodies responsible for approving such decisions. The quality of published research plays a significant role in each of these stages, especially in research-intensive universities, but nowhere is it the sole criterion. The quality of teaching, administration and external engagement all play a role. Nonetheless, in many universities research is the most important criterion, and judgments are often mapped closely on the approach of the REF. This is for obvious reasons, given the importance of RAE/REF decisions for departmental and institutional reputation and income.

14What has been the impact since 1986 of this process of evaluation on research itself? It has crucially underpinned the significant growth of government funding for research since 1997: its growth under the New Labour government and its relative protection under the Coalition government. The use of new accounting methods to establish the full economic costs of project research funded by the UK’s seven Research Councils was one key element in this funding growth, while the evaluation of outputs through the RAE/REF provided assurance to governments that, for the funding that came as a block grant through the higher education funding councils, that quality was being maintained and enhanced in international comparison.

15What about its effect on the character of the research itself? Universities plan for several years whom to submit to the RAE/REF (leaving some decisions until the last moment), which of their outputs to submit, and whom to recruit to the staff to improve the submission. HEFCE has established strong guidelines to ensure that the process meets equal opportunity requirements, but it is hard to ensure that these are not overlooked in a process that is increasingly managerially driven. Each university will also actively recruit staff who they think will improve their submission. This movement of staff is often seen as being a damaging development but it introduces an element of dynamism into the system that might not otherwise exist. Universities do not always get their calculations right, but it broadly leads the strongest researchers to gravitate towards the strongest research departments. That may be bad for research distribution across the system, but it is probably good for the quality of UK research as a whole.

  • 5 I am currently leading a quite separate project for HEFCE on the monograph and open access.

16The RAE/REF may have raised the overall quality of UK research over the last 30 years, but has it also narrowed how researchers disseminate their results? The answer must be ambivalent. By opening up new forms of research output—from art exhibitions to reports to industry, from musical performances to designs—it has helped integrate some disciplines within a research environment. Although some claim that it has led to journal articles being privileged over monographs, there is no evidence from the RAE/REF submissions that that is the case, and the pressures on monograph publishing come from other sources.5 Publishing in high-quality journals has become even more competitive. Although submitted publications may be in any language, there is a perceived pressure to publish in English, which is particularly limiting in the humanities. The two greatest criticisms of the impact of research evaluation is that it has been a disincentive to risky research which might be less likely to yield high-quality publications, and that it militates against interdisciplinary research which tends to be published in less prestigious journals and where discipline-based peer review might inhibit appreciation. Notwithstanding the efforts of HEFCE to establish criteria and processes that limit those threats, they undoubtedly remain in the minds of university management and of researchers themselves. But then any system of evaluation changes and affects the phenomenon that is being evaluated, and this inevitably applies to research publications.

Inicio de página

Notas

1  G. Crossick is Distinguished Professor of the Humanities in the School of Advanced Study, and was previously Vice-Chancellor of the University of London (2010-2012).

2 For full details of the methods used by the main panels and sub-panels, see HEFCE, REF 2014. Panel criteria and working methods, Bristol 2012.

3 Development of Bibliometric Indicators of Research Quality. A Report to HEFCE by the Centre for Science & Technology Studies, Leiden University (Leiden, 2008).

4  HEFCE, Ref 2014, p. 25.

5 I am currently leading a quite separate project for HEFCE on the monograph and open access.

Inicio de página

Para citar este artículo

Referencia en papel

Geoffrey Crossick, «Evaluation of research publications and its impact on research: the case of Great Britain»Mélanges de la Casa de Velázquez, 44-2 | 2014, 287-291.

Referencia electrónica

Geoffrey Crossick, «Evaluation of research publications and its impact on research: the case of Great Britain»Mélanges de la Casa de Velázquez [En línea], 44-2 | 2014, Publicado el 01 enero 2018, consultado el 29 marzo 2024. URL: http://journals.openedition.org/mcv/5856; DOI: https://doi.org/10.4000/mcv.5856

Inicio de página

Autor

Geoffrey Crossick

University of London

Inicio de página

Derechos de autor

CC-BY-NC-ND-4.0

Únicamente el texto se puede utilizar bajo licencia CC BY-NC-ND 4.0. Salvo indicación contraria, los demás elementos (ilustraciones, archivos adicionales importados) son "Todos los derechos reservados".

Inicio de página
Buscar en OpenEdition Search

Se le redirigirá a OpenEdition Search