Developing a Model for Evaluating the Effectiveness of Academic Research: A Meta-Synthesis Study

Document Type : Review Article

Authors

1 Professor of Education Economics, Faculty of Psychology and Education, University of Tehran, Tehran, Iran.

2 Ph.D. Student, Higher Education Economics, Faculty of Psychology and Education, University of Tehran, Tehran, Iran.

3 Professor, Biophysics Department, Institute of Biochemistry and Biophysics, University of Tehran, Tehran, Iran.

4 Associate Professor of Higher Education Administration, Shahid Sattari Aeronautical University of Science and Technology, Tehran, Iran.

Abstract

The main goal of the present research was to design a model for evaluating the effectiveness of university research, which was achieved using the meta-synthesis method. First, with the aim of identifying the dimensions, criteria and indicators of the effectiveness of academic research, selected keywords were searched in the most important domestic and foreign reliable academic journals and databases and the number of 262 related studies was identified and extracted among which 30 related scientific studies were selected according to using certain criteria. In the next step, with applying the method of content analysis, the related codes and concepts were extracted and then criteria and indicators related to each of the dimensions were identified and the findings were categorized upon them. Then, the key findings under the three dimensions of "inputs, outputs and effectiveness (internal and external aspects)" of academic research were compiled. The results showed that this proposed multidimensional and integrated model is unique and can be considered as an effective tool for evaluating the effectiveness of academic research at the disposal of researchers, policy makers and practitioners. Its application and implementation can improve the quality of decision-making process as well as the resource allocation decisions at different operational levels of the university.

Keywords

References

  1. Abramo, G., D’Angelo, C., & Pugini, F. (2008). The measurement of Italian universities’ research productivity by a non-parametric-bibliometric methodology. Scientometrics, 76(2), 225-244.
  2. Altbach, P. G., Reisberg, L., & Rumbley, L. E. (2010). Trends in global higher education: Tracking an academic revolution. Rotterdam: Sense Publishers.
  3. Avkiran, N. K. (2001). Investigating technical and scale efficiencies of Australian universities through data envelopment analysis. Socio-economic planning sciences, 35(1), 57-80.
  4. BabaAkbari Sari, A., Ghagremani, M., Fathivajargah, Kuroush., & Motmeni, A. (2017). Evaluating the outputs, effects and consequences of management research in 2013. Public Administration Perspective, 8(3), 41-65. {In Persian}
  5. Bandola-Gill, J. (2019). Between relevance and excellence? Research impact agenda and the production of policy knowledge. Science and Public Policy, 46(6): 895-905.
  6. Banzi, R., Moja, L., Pistotti, V., Facchini, A &, Liberati, A. (2011). Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res. Policy Syst., 9(1), 26-36.
  7. Bazargan, Abbas. (2018). Evaluation in higher education. Tehran: The Center for Studying and Compiling University Books in Humanities (SAMT). {In Persian}
  8. Chao, S. M., & Chen, M. J. (2023). DEA Approach to Evaluate Research Efficiency of Departments in University. Engineering Proceedings, 38(1), 71.
  9. Ding, T., Yang, J., Wu, H., Wen, Y., Tan, C., & Liang, L. (2021). Research performance evaluation of Chinese university: a non-homogeneous network DEA approach. Journal of Management Science and Engineering, 6(4), 467-481.
  10. Ehsani, V., Azami, M., Najafi, S. M. B., & Soheili, F. (2017). The effectiveness of domestic Scientific research on Iran development Indicators. Iranian Journal of Information Processing and Management, 32(2), 319-347. doi: 10.35050/JIPM010.2017.044 {In Persian}
  11. Eslami, Z., Hakimzadeh, R., Saboury, A. A., & Farzad, V. (2021). Identifying effective factors in the assessment of research productivity of faculty members in the Humanities and Social Sciences (Case Study: University of Tehran). Journal of Management and Planning in Educational System, 13(2), 149-176. doi: 10.52547/mpes.13.2.149 {In Persian}
  12. Etzkowitz, H. (2004). The evolution of the entrepreneurial university. International Journal of Technology and Globalization, 1(1): 64-77.
  13. Farazkish, M., & Dastranj, N. (2019). Selecting and applying science, technology and innovation evaluation indices. Journal of Science and Technology Policy, 12(2), 579-598 {In Persian}
  14. Feizabadi, M., Fahimnia, F., Naghshineh, N., Tofighi, S., & Mosavi -Jarrahi, A. (2017). A review of impact assessment indicators of medical research. Health Information Management, 13(6), 432-437. {In Persian}
  15. Ghagremani, M., Fathivajargah, Kuroush., & Motmeni, A. (2017). Evaluating the outputs, effects and consequences of management research in 2013.Public Administration Perspective, 8(3), 41-65 {In Persian}
  16. Ghazinoory, S., & Farazkish, M. (2018). A modal for STI national evaluation based efficiency, effectiveness and utility index. Strategic Studies of Public Policy, 8(27), 205-229. {In Persian}
  17. Ghazinoory, S., Farazkish, M., Nasri, S., & Mardani, A. (2023). Designing a science, technology, and innovation (STI) evaluation dashboard: A comprehensive and multidimensional approach. Technology Analysis & Strategic Management, 35(8), 1005-1023.
  18. Ghomi, H. R., Rahmani, M., & Khakzar, M. (2017). Measuring the research performance of Iranian State University through utilizing Analytic Hierarchy Process and Network Data Envelopment Analysis. MILITARY MANAGEMENT QUARTERLY, 17(67), 141-167. {In Persian}
  19. Golhasany, H., Hosseini, T., & Hassanzadeh, M. (2021). Challenges to creating impact in humanities and social sciences in Iran: A Grounded Theory Analysis. Sciences and Techniques of Information Management, 7(2), 141-174. doi: 10.22091/stim.2020.5770.1415 {In Persian}
  20. Graham, K. E., Chorzempa, H. L., Valentine, P. A., & Magnan, J. (2012). Evaluating health research impact: Development and implementation of the Alberta Innovates–Health Solutions impact framework. Research Evaluation, 21(5), 354-367.
  21. Graham, I.D.; McCutcheon, C. & Kothari, A. (2019). Exploring the frontiers of research coproduction: The Integrated Knowledge Translation Research Network concept papers. Health Research Policy and Systems, 17(1): 88.
  22. Hinrichs-Krapels, S., & Grant, J. (2016). Exploring the effectiveness, efficiency and equity (3e’s) of research and research impact assessment. Palgrave Communications, 2(1), 1-9.
  23. Hu, Q., & He, W. (2011). Evaluation of scientific research efficiency of 985 program universities based on DEA. Science & Technology Progress and Policy, 28(19), 135-139.
  24. Joly, P. B., Gaunand, A., Colinet, L., Larédo, P., Lemarié, S., & Matt, M. (2015). ASIRPA: A comprehensive theory-based approach to assessing the societal impacts of a research organization. Research Evaluation, 24(4), 440-453.
  25. Kashim, R., Kasim, M. M., & Rahman, R. A. (2017). Measuring effectiveness of a university by a parallel network DEA model. In AIP Conference Proceedings (Vol. 1905, No. 1, p. 040014). AIP Publishing LLC.
  26. Khajevand, Z., Afshin., Z. (2016). Productivity measurement and ranking of research units using Data Envelopment Analysis. Basparesh, 5(4), 92-99. doi:10.22063/basparesh.2016.1228 {In Persian}
  27. Khosravi, M., & Pournaghi, R. (2019). Dimensions of research impact: A systematic review. Scientometrics Research Journal, 5(1, Spring & Summer), 203-224. doi: 10.22070/rsci.2018.695 {In Persian}
  28. Kuah, C. T., & Wong, K. Y. (2011). Efficiency assessment of universities through data envelopment analysis. Procedia computer science, 3, 499-506.
  29. Kuruvilla, S., Mays, N., Pleasant, A., & Walt, G. (2006). Describing the impact of health research: A research impact framework. BMC health services research, 6, 1-18.
  30. Li, Q. B., & Ren, Z. X. (2009). Empirical analysis on scientific research efficiency of Chinese provincial universities 2002–2006: Based on the DEA model. Journal of Shanxi University of Finance and Economics (Higher Education Edition), 1(001).
  31. Meng, W., Zhang, D., Qi, L., & Liu, W. (2008). Two-level DEA approaches in research evaluation. Omega 36(6): 950-957.
  32. Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, 24(4), 405-419.
  33. Naderi, Abolghasem. )2018). Advanced topics in economics of education: Internal efficiency and effectiveness. Tehran: University of Tehran. {In Persian}
  34. Newson, R., King, L., Rychetnik, L., Bauman, A. E., Redman, S., Milat, A. J., Schroeder, J., Cohen, G., & Chapman, S. (2015). A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: Perceptions of Australian researchers. BMJ Open, 5(7), 1-14.
  35. Noroozi Chakoli, A., & Abdi, S. (2020). Challenges, issues and structural require elements for the implementation of the national system for evaluating the effectiveness of Science, Technology and Innovation. Rahyaft, 30(77), 89-106. doi: 10.22034/rahyaft.2020.13821 {In Persian}
  36. Pettigrew, A.M. (2011). Scholarship with impact. British Journal of Management, 22(3): 347-354.
  37. Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm, P., & Primeri, E. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308.
  38. Rhaiem, M. (2017). Measurement and determinants of academic research efficiency: a systematic review of the evidence. Scientometrics, 110(2), 581-615.
  39. Roy, S., Nagpaul, P. S., & Mohapatra, P. K. (2003). Developing a model to measure the effectiveness of research units. International Journal of Operations & Production Management, 23(12), 1514-1531.
  40. Sandelowski, M, and Barroso, J (2007). Handbook for synthesizing qualitative research. Springer: New York.
  41. Tayefehbagher, D., Abazari, Z., Moradi, S., & Babalhavaeji, F. (2022). A model for evaluating the impact of humanities research in Iran. Scientometrics Research Journal, 8(1, Spring & Summer), 1-34. doi: 10.22070/rsci.2020.13384.1450 {In Persian}
  42. Torkzadeh, J., Sadeghian Sourki, M., & Aghili, R. (2019). Continuing the patterns and framework for evaluation and audit of academic research: Conceptual approach. Public Policy in Administration, 10(1), 105-119. {In Persian}
  43. Glossary Terms. (2020). www.usq.edu.au/planstats/Docs/Glossary Terms.doc.579–598.
  44. Vosough, F., Ahmadi, G., Hassanmoradi, N., & Mohammaddavoodi, A. (2021). Identifying the native dimensions, components and indicators of research performance evaluation in university (A case study: Islamic Azad University in Tehran). Journal of Management and Planning in Educational System, 14(1), 13-42. doi: 10.52547/MPES.14.1.13 {In Persian}
  45. Weiss, Anthony P. "Measuring the impact of medical research: moving from outputs to outcomes." American Journal of Psychiatry 164, no. 2 (2007): 206-214.
  46. Williams, K. & Grant, J. (2018). A comparative review of how the policy and procedures to assess research impact evolved in Australia and the UK. Research Evaluation, 27(2): 93-105.
  47. Williams, V. L., Eiseman, E., Landree, E., & Adamson, D. (2009). Demonstrating and communicating research impact: preparing NIOSH Programs for External Review. Santa Monica: RAND Corporation.
  48. Xu, J. (2009). An evaluation of relative input–output efficiency of research in higher education institutions in China: A data envelopment analysis (DEA). Tsinghua Journal of Education, 30(2), 76-80.
  49. Zhang, L., Bao, W., & Sun, L. (2016). Resources and research production in higher education: A longitudinal analysis of Chinese universities, 2000–2010. Research in Higher Education, 57(7), 869-891.
Iranian Journal of Public Policy
Volume 9, Issue 4
March 2024
Pages 97-114

Files

Share

How to cite

Statistics