When shaping the methods and allocation of research funding, it’s also worth stopping to reflect on the positive implications of the new arrangements. A new type of strategic research funding has been distributed since 2015. From the outset, the projects and programmes have described their activities and successes widely in public. The Strategic Research Council (SRC) and the officials of the Academy of Finland supporting it have also sought transparency and widely explained their actions and the underlying justifications. However, the first comprehensive responses have only been received in the past year, after the first programme evaluations of strategic research were completed. What are the results of the evaluations?

Expectations for the programme evaluation

What should the programme evaluation of strategic research focus on? Who would the evaluation information be useful for? How could the expectations imposed on the evaluation be met with the resources available? Such questions were asked when consideration of the implementation of the programme evaluation started in 2018. The first programme evaluation focused on four SRC programmes ending in autumn 2019. Compared to other SRC programmes, these programmes had an exceptional funding period of three years.

After the preparatory work in spring 2019, the SRC recorded in its funding principles that the programme evaluation should focus specifically on assessing the programmes’ societal impact. At the same time, it was suggested that the evaluation results should be relevant to both the scientific community and representatives of government bodies. The results should also benefit strategic research projects in the future and the work of the SRC. As a result, the main objective ended up being to understand the diversity and societal impact of strategic research activities: after all, SRC programmes were all about bringing together multidisciplinary research and stakeholders in a completely new way, and there were no straightforward indicators in place for its evaluation. And so, it was necessary to first learn to understand what is valuable in strategic research activities and why. Only then would an evaluation follow.

The evaluation question concerning all research programmes ended up addressing how the research and interaction work done in projects and programmes affected other parties. Did the actors playing a significant role in resolving societal challenges obtain a new scientific basis for their activities, or change their operations to overcome the challenges at hand? Researchers and projects were understood as enablers of impact, since the introduction of societal changes depends extensively on different societal actors. However, researchers and research projects were regarded as responsible for making societal impact possible: projects were expected to provide new research data to support social decision-making and to disseminate information in ways that support the utilisation of information as well as possible. In line with the objectives of strategic research, high-quality research that is innovative in its multidisciplinary approach was seen as an essential means of supporting the use of information.

Since this was a new funding instrument, which meant that only the actors in the research programmes had first-hand knowledge of their activities, it made sense to invite the projects and their managers to participate in the evaluation. The self-assessment was followed by an evaluation of what societal impact the programmes had achieved or enabled. This was completed by Gaia Consulting, a consultancy company familiar with Finnish society and the scientific community. Finally, the scientific activities and results were assessed in evaluation panels consisting of international experts.

Increased understanding and advanced operating models

The evaluations indicate that the strategic research programmes have, through their versatile activities, established strong foundations for the creation of societal impact. The effects of research and interaction work include an increased understanding and societal debate on the key phenomena around which the research programmes were built. In addition to increasing understanding, the projects succeeded in producing concrete practical operating models and solutions to meet the challenges identified in the programmes. A key method was co-creation with information users and other stakeholders. The evaluation also recognised projects’ efforts to disseminate research-based information to wider target groups through policy briefs and clear and comprehensible publications, to mention just a few examples.

It’s obvious, however, that verifying the societal impact of large-scale research programmes is challenging, and concrete examples may only be detectable far in the future. And so, in terms of the programmes’ impact, it may be more straightforward to talk about the stakeholder networks created and the development of their activities rather than actual societal changes. The programme evaluation also contributed to a better understanding of how challenging it is to appraise the success of multidisciplinary research projects and their stakeholder work.

Impressive multidisciplinary research activities

The projects in the research programmes were required to be multidisciplinary. According to the programme evaluation, this was reflected not only in the composition of the projects, but also in the objectives, methods and results of research and interaction work. Through multidisciplinary cooperation, it was considered that the programmes had led to, amongst other things, new understanding and concepts between disciplines, and new methodological openings that bring together different disciplines. According to the evaluation, the research programmes also had a wide range of research and interaction methods and objectives. The societal phenomena behind the programmes and the starting points for each research field played a role in the choices made. In terms of these methods and objectives, there were considerable differences between the research programmes.

The importance of multidisciplinary cooperation is best demonstrated in its institutional impact. Through strategic research funding, researchers have been able to create new multidisciplinary researcher networks and interaction. According to the findings of the programme evaluation, this creation of cooperation and interaction required a wide range of resources, and different alternative costs had to be taken into account in the planning of activities.  As a rule, strategic research programmes continue for six years. On the other hand, it was stated in the self-evaluations that even three-year funding gave programme actors more opportunities to focus specifically on multidisciplinary planning of research and interaction activities.

Programmes as implementers of responsible research

The principles of responsible research were reflected in the activities of the programmes and projects in the openness of scientific publications and in enabling further use of data. Enabling access to data for further processing was highlighted in the evaluation as an important area for development, as it was considered to be one of the most concrete ways to promote the continuity of research after the programme period. The programme evaluation also recognised examples of how taking the principles of responsible science into account lays the foundation for the achievement of societal impact objectives. Through the methods of multidisciplinary co-creation, it was possible to integrate groups in a weak societal position into the research processes. This helped include the views of vulnerable groups as part of the research content.

How can the evaluation process be developed and how can evaluations develop strategic research?

For the most part, the programme evaluation was considered successful. A diverse consultation with interested parties and evaluators created a balanced and extensive picture of the topic under review. At all stages of the evaluation process, it could be observed that the scientific community and societal actors were very interested in the evaluation and the lessons learned from it.

Of course, challenges were also encountered when organising the evaluation. One of them was the scope of the evaluation material. It posed challenges to both the organisation of the evaluation process and the completion of the assessment task itself. Project monitoring during the activities produced a large amount of indicator data and free-form reflective material, the management of which was challenging for anyone. However, the experts who carried out the evaluation often wondered whether relevant information and material would still have been available somewhere to describe some aspect of the programme activities in a broader or more in-depth manner.

The diverse objectives of the evaluation work made it difficult to see the big picture. This was not made any easier by the fact that the programmes under review differed from each other significantly in terms of both their content and the number of projects. The most important general observation was that, from the perspective of the evaluation work, the programmes would have needed clearer objectives. At the same time, however, it became obvious that setting concrete impact objectives may not necessarily be sensible or possible in the constantly changing society, in which changes in political power relations may drastically change the handling of many different societal challenges.

The programme evaluation was carried out in a period of just over a year after the research programmes’ funding period had ended. The timing gave the programme actors an opportunity to reflect on the completed research and interaction activities immediately after the funding period had ended. On the other hand, the timing created challenges for evaluating the results and, in particular, the effects and impact. The objectives of research and interaction activities are far-reaching. Accordingly, it is clear that neither scientific nor societal impact could be verified in an unambiguous and definitive manner on the basis of this evaluation. In the future, these framework conditions will also play a significant role in the organisation of programme evaluations and setting of evaluation objectives.

The question arises whether it is worth carrying out the programme evaluations systematically at the end of each research programme? The evaluation method now implemented can be used to create a reliable understanding of the success of the programme at a specific point in time. On the other hand, a regular and real-time review of the activities of finished and ongoing programmes could provide a more comprehensive understanding of the impact of strategic research, and even strengthen the ability to support the programmes better in the future.

The programme evaluation and its results have attracted wide interest. The Strategic Research Council and the Academy Division supporting it aim to continue the valuable exchange of information and learning on the evaluation of multidisciplinary and societally high-impact research.