When discussing evaluations, a distinction should be made between the evaluation of research centres funded by the REI and the evaluation of the funding scheme itself. Although the two are related, the latter usually includes questions about the REI’s impact on the national science system, and possibly the “added value” of the REI as compared to other modes of funding. For the REIs discussed here, such evaluations have been less frequent than evaluations of individual centres, which are often an obligatory aspect of the scheme. Moreover, many REIs have not been running long enough for comprehensive long-term evaluations. Nevertheless, several countries reported having conducted (interim) evaluations of their REIs. Table 2.5 provides an overview of the survey responses.
Table 2.5. REI system evaluations
An expert panel produces annual reports on the performance of the centres and the wider effects of the REI on the research system.
Evaluations of the CoE programme are carried out with a specific thematic focus. The most recent evaluation focused on the societal impacts of the CoE programmes in 2000-05 and 2002-07.
Commissioned by the German Research Foundation (DFG), the Institute for Research Information and Quality Assurance (iFQ) implemented a monitoring system that estimates both the intended and unintended effects of the REI on the basis of quantitative and qualitative empirical data. An evaluation report based on these data was written in 2008. A report on the second funding cycle is scheduled for 2015. A systematic evaluation of the REI by international experts will take place in 2016.
Germany - Hesse
The REI was evaluated by the German Research Council in 2012.
An evaluation of the REI system was carried out by the Wissenschaftszentrum Sachsen-Anhalt (Science Centre Saxony-Anhalt, WZW).
The REI was assessed in 2008, and another evaluation was carried out in 2012. The results of the evaluation were used to inform policy.
An impact assessment of funding cycles 1-3 was conducted by PA Consulting for the Irish Higher Education Authority in 2011. The report focuses on the direct commercial and economic impacts of the scheme.
Policy evaluations are conducted every year.
In 2010, the Royal Netherlands Academy of Arts and Sciences (KNAW) reviewed the REI as a whole.
The REI was evaluated by the Nordic Institute for Studies in Innovation, Research and Education (NIFU) in 2010.
In 2010, the American Association for the Advancement of Science (AAAS) produced a review of the STC programme (2000-09).
Source: OECD/RIHR questionnaire to government ministries, Q5.4: Has the REI funding system as a whole been evaluated? (please describe).
the STC programme (2000-09).
The respondents were also asked to provide an assessment of whether the national REI(s) had achieved its (their) goals. Overall, 22 REIs were reported to have achieved their goals. For a further six it was too early to tell. No respondents answered that goals were not achieved. The following summarises answers from ministries that discussed goal achievement and wider effects of REIs:
• Denmark, UNIK: The REI has led to a new culture of collaboration within host universities, but also with national and international network partners. The UNIK universities have strengthened their international competitiveness, e.g. through infrastructure improvements supported by the REI. UNIK centres had an influence on policy deliberations leading up to the European Union’s eighth Framework Programme on Research and Technological Development, Horizon 2020.
• Estonia, CoE: The REI has helped to improve the competitiveness of Estonian R&D, as shown by growing success in competitive applications to the EU’s research funding programmes.
• Finland, CoE: The development of creative research and researcher training environments has progressed greatly. The accumulated knowledge and know-how of funded centres has disseminated into the national research system. The scheme has promoted the development of priority areas in HEIs.
• Germany, Excellence Initiative: The research output of REI-funded centres has increased significantly in quantitative and qualitative terms. Knowledge transfer to industrial and medical applications has been successful. The REI’s Graduate Schools are setting standards for recruiting, supervision and study planning. New paths of academic qualification have opened up. Inter-institutional co-operation that was previously not practicable has been established. Universities have furthered their structural modernisation.
• Germany/Saxony-Anhalt, Networks of Scientific Excellence: Research quality has improved in various respects, and collaboration between researchers and between research projects has increased and resulted in more concentration of research and innovation capacity. Internationally visible research environments have been successfully established.
• Germany/Thuringia, ProExcellence: New and productive research structures have been created. Novel interdisciplinary approaches have been adopted and have a positive effect on science, teaching and economic development. Universities and research facilities are able to compete for competitive national and European funds. Young researchers supported through the REI have performed particularly well.
• Ireland, CSET: Along with similarly structured schemes run by Science Foundation Ireland, CSET has increased interaction between academics and industry and the attractiveness of Irish universities as research partners. The scheme has improved the reputation of the Irish research base and increased Ireland’s attractiveness as a location for industrial R&D. In host institutions, CSET has instigated an increased emphasis on research strategy, management, collaboration and commercialisation in both strategic and administrative terms.
• Ireland, PRTLI: The scheme has led increased research outcomes (publications, citations, etc.). The increase in human capital for research is dispersing through industry and the public sector. HEIs now have a more strategic and planned approach to the long-term development of their research capabilities. The quality and relevance of graduate output and skills at all levels have been enhanced. High-quality infrastructure and capacity support for outstandingly talented researchers and teams within institutions are now in place. More co-operation by researchers within and between institutions, including inter-institutional cooperation, now exists. Significant private and international funding has been leveraged as a result of PRTLI.
• Japan, Global COE/WPI: Training of doctoral candidates has been improved. Centres have closed the gap on the world’s best research environments, as shown by international third-party funding, applications for positions in centres from around the world, and publications of the highest quality.
• Korea, BK21/WCU: Early-stage research has been fostered, research quality has improved, and better linkages with industry have been established. International collaborative research has been strengthened, and the international visibility of Korean universities has increased.
• Netherlands, BIS: The scheme has led to a higher concentration of top researchers in BIS centres and better research output.
• New Zealand, CoRE: There have been improvements in research quality, as measured by outputs such as citation rates and postgraduate completion rates or external research income. Host institutions have raised research quality through clearer funding incentives.
• Norway, CoE/CRI/CEER: The CoE scheme has been particularly successful in terms of promoting researcher recruitment, strengthening the internationalisation of Norwegian research and, for CoE and CRI, creating networks through national and interdisciplinary collaboration. The competitive nature of all three REIs has helped to raise the standards of research in general. The schemes have led to a stronger focus on scientific leadership and strategies. Both CRI and CEER have increased academy-industry partnerships. CRI has encouraged application- oriented research that has benefited supporting industries and organisations in the public sector by providing innovative ideas for enhancement of processes and product development.
• Portugal, Multi-Year Funding Programme: Increased research quality and quantity throughout the Portuguese research system, as measured by the number of publications, are assumed to be linked to the REI and the programmes associated with it.
• Slovenia, Centres of Excellence: The scheme has improved capacity for support to young researchers. Concentration and specialisation have been successfully initiated.
• Sweden: An exhaustive study of effects of the Swedish REIs has yet to be carried out, but in view of earlier experiences with similar funding schemes, REIs are expected to contribute significantly to new approaches to interdisciplinary and innovative research, to closer academia-industry relationships, and to new and effective forms of leadership. An evaluation of the Berzelii Centres has revealed numerous strengths characteristic of the scheme, e.g. coherent basic and applied research strategies aligned with the vision and mission of the centres, successful development of young researchers, transfer of research outcomes, and collaboration between disciplines and between universities and the business sector.
• United States, STC: The centres have encouraged the development of new technologies, new instrumentation, and new approaches to pressing societal challenges; have resulted in broader collaboration; and have an excellent record in producing new master’s and PhD students.
In sum from the point of view of most ministries and funding bodies, REIs have had an overwhelmingly positive effect. It is worth noting that these stakeholders assess the success of REIs not only in terms of immediate scientific outputs (e.g. publications), but also, and very prominently, in terms of the centres’ contribution to linkages between disciplines, institutions and sectors. This suggests that besides their support to selected research units, REIs are as important drivers of collaboration and horizontal linking.
When asked about the unintended effects of REI(s), respondents reported positive experiences for the most part, e.g. improvements in strategic management in HEIs, or unexpected but fruitful forms of co-operation. Two respondents reported an unintended effect which they considered rather negative, i.e. the fact that some disciplines were overrepresented in the final selection when compared to the range of eligible fields. In these cases the natural and life sciences appear to have met the application requirements better than other fields. In one case it was noted that requirements for the host institution to contribute funds to the REI centre(s) might come at the expense of other research projects or groups that would deserve funding from their institution but are not involved in an REI scheme.