Balancing diversity and standardization in research out-put indicators
- Focusing on the humanities and social sciences -

HAYASHI Takayuki, Professor, National Graduate Institute for Policy Studies
FUJIMITSU Chika, Japanese Government Long-term Fellow, Personnel Division Minister's Secretariat, Ministry of Education, Culture, Sports, Science and Technology
HATA Yusuke, Unit Chief, Planning and Evaluation Division, Science and Technology Policy Bureau, Ministry of Education, Culture, Sports, Science and Technology
NAKAWATASE Hidekazu, Project Researcher, National Institute of Informatics
ANDO Nika, Program Specialist, Science for RE-Designing Science, Technology and Innova-tion Policy Center (SciREX Center) National Graduate Institute for Policy Studies

June 2021

SciREX Center

SciREX-WP-2021-#02

SciREX-WP-2021-#02_Hayashi.pdf

Research outputs are measured for a variety of purposes, including funding decision and organizational strategy formation. One thing to keep in mind when measuring research outputs is that although there is no one-fit-all set of indicators for all research fields, a limited number of indicators are likely to be used. It causes undesired effects on a researcher’s behavior. This working paper examines issues related to various research output indicators, focusing mainly on the humanities and social sciences (SSH).

In the context of "performance-based funding allocation" to universities in Japan, it is required to set a small number of comparable standard indicators. However, the academic community has pointed out the importance of recognizing as many different types of research results as possible in order to encourage diverse research activities, although it is difficult to tabulate them in a comparable manner. Therefore, arguments don't overlap each other. Balancing "diversity" and comparable "standardization" is an issue in actual institutional design. This balancing is dealt with differently between a metrics-centered evaluation and a peer-review-centered evaluation. As a result of analyzing the situation overseas, as a metrics centered evaluation, a method called the Norwegian model is adopted, which measures "academic publications" that are defined including those other than English journal articles. For this model, there is a need for a domestic database, a scholarly community to define "academic publications" and academic publishing channel lists, and a mechanism for monitoring the impact of measurements. On the other hand, in a peer-review-centered evaluation, it is necessary to broadly set the definition of research, and to create examples of various research outputs in each field by evaluation institutions, etc., and to standardize the definition and description for them.

Regarding the possibility of implementing such efforts in Japan, we analyzed the submitted research outputs for university evaluations and the reports of the Grant-in-Aid for Scientific Research focusing on history and management research from SSH and compared them with the achievements at REF2014 in the UK. As a result, the variety of research outputs in Japan is higher than in the UK, and there is a tendency to publish outputs for a wider audience, not classifying journals and publishers strictly as "academic publications". Thus, It was suggested that it is currently difficult to use the defined "academic publication" measurement as in overseas. In the measurement of social impact of research which has not been sufficiently established internationally, the general issues regarding the diversity and standardization are also discussed.

There are several points to keep in mind in balancing diversity and standardization in both the evaluation system. It is expected that stakeholders such as universities and academies will discuss not only measurement but also fundamental discussions on the value of research in SSH. Furthermore, as "converging knowledge" including SSH is required to promote social transformation in the future, it is necessary to consider how to evaluate it based on these issues.