Evaluation stages

Evaluations of disciplines and individual fields of research are an important research and science policy development tool. Their main purpose is to provide feedback to the scientific community and to funding agencies. Furthermore, they provide an opportunity for learning and development for all those involved. Evaluations inspire discussion and debate and help researchers and funding organisations to identify potential problems and areas of development. They have a whole array of effects: conscious, unconscious, positive, negative, intended and unintended as well as short-term and long-term effects.

Implementation of evaluations

The Academy is widely recognised for its activity in the evaluation of individual disciplines and fields of research. The initiative for such an evaluation may come from the research community, Research Councils, other science funding agencies or the authorities. An evaluation often marks the beginning of or follows up a development effort in a certain field of research.

A steering group is appointed to take charge of planning and implementation of evaluations. The coordinator is almost always appointed from outside the Academy or the Academy’s Administration Office.

The purpose and objective of evaluation is to gain an expert and independent view of the state of the discipline in question, of the quality of its research and its development needs. The researchers and experts appointed to the evaluation panel come predominantly from outside Finland. All experts are well-respected names in the field concerned, and they are expected to have good experience in science policy. The chair of the evaluation panel has a crucial influence on the success of the evaluation. The chair shall be an internationally respected scholar and have sufficient independence vis-à-vis the subjects of evaluation.

Feedback from international experts is key to discipline evaluations
Earlier discipline evaluations have recommended that coordination be increased among research units in the field in question; that research funding be allocated in the future to fields that benefit most from the input; and that the size of research units be increased so that broad and complex research problems can be addressed and resolved. The panels have also observed that, in some fields, the mean age of graduating PhDs is still relatively high and that steps are needed to encourage greater mobility among researchers. One important use of discipline evaluations is in the preparation of new Academy research programmes.

Aims of evaluations

The purpose of evaluations is to

• provide information on the international standing and quality of Finnish research
• identify the strengths and weaknesses of the discipline or research field in question and to explore the underlying reasons
• identify development needs and prepare recommendations for researchers, research organisations and funding bodies on how to raise the quality of research
• provide information on the societal, technical and economic impacts of research
• help identify the weaknesses and development needs of Finnish research and the Finnish research system
• increase awareness of researchers and their achievements both in Finland and abroad

Evaluation criteria

The criteria for evaluations are designed according to the grounds and focus of each evaluation. The criteria applied may include:

• originality of the subject and research problems
• creativity and topicality
• appropriateness of scientific methods
• scientific visibility: impact factors of publication forums, citations received by publications or researchers
• organisation and effectiveness of researcher training in the field concerned
• systematic promotion and internationalisation of postdoctoral research careers
• international recognition of researchers: significant science awards, major positions of trust in scientific associations, editorial positions in scientific publications, etc.
• societal impacts, e.g. various positions of expertise and contribution to science popularisation

Development of evaluations

Every evaluation is a picture of an arrested moment, an assessment of the situation at the time of the evaluation that also makes use of retrospective data. However, what is most important of all is what happens in the future, because research is in a constant state of flux.

The changes taking place in the nature of research constantly throw up new challenges to traditional discipline evaluations. Work to develop and improve these evaluations will continue to gain in significance as research funding bodies are expected to step up their contribution to assessing the societal impacts of research. In the new and changing operating environment, new evaluation procedures are needed that take account of 1) the interaction between basic and applied research, 2) the impact of funding for high-risk, breakthrough research, 3) the different resources of cutting-edge research and new emerging disciplines, and 4) research on the interface of different disciplines.