research analysis and evaluation

• Privacy Policy

Home » Evaluating Research – Process, Examples and Methods

Evaluating Research – Process, Examples and Methods

Table of Contents

Evaluating Research

Definition:

Evaluating Research refers to the process of assessing the quality, credibility, and relevance of a research study or project. This involves examining the methods, data, and results of the research in order to determine its validity, reliability, and usefulness. Evaluating research can be done by both experts and non-experts in the field, and involves critical thinking, analysis, and interpretation of the research findings.

Research Evaluating Process

The process of evaluating research typically involves the following steps:

Identify the Research Question

The first step in evaluating research is to identify the research question or problem that the study is addressing. This will help you to determine whether the study is relevant to your needs.

Assess the Study Design

The study design refers to the methodology used to conduct the research. You should assess whether the study design is appropriate for the research question and whether it is likely to produce reliable and valid results.

Evaluate the Sample

The sample refers to the group of participants or subjects who are included in the study. You should evaluate whether the sample size is adequate and whether the participants are representative of the population under study.

Review the Data Collection Methods

You should review the data collection methods used in the study to ensure that they are valid and reliable. This includes assessing the measures used to collect data and the procedures used to collect data.

Examine the Statistical Analysis

Statistical analysis refers to the methods used to analyze the data. You should examine whether the statistical analysis is appropriate for the research question and whether it is likely to produce valid and reliable results.

Assess the Conclusions

You should evaluate whether the data support the conclusions drawn from the study and whether they are relevant to the research question.

Consider the Limitations

Finally, you should consider the limitations of the study, including any potential biases or confounding factors that may have influenced the results.

Evaluating Research Methods

Evaluating Research Methods are as follows:

• Peer review: Peer review is a process where experts in the field review a study before it is published. This helps ensure that the study is accurate, valid, and relevant to the field.
• Critical appraisal : Critical appraisal involves systematically evaluating a study based on specific criteria. This helps assess the quality of the study and the reliability of the findings.
• Replication : Replication involves repeating a study to test the validity and reliability of the findings. This can help identify any errors or biases in the original study.
• Meta-analysis : Meta-analysis is a statistical method that combines the results of multiple studies to provide a more comprehensive understanding of a particular topic. This can help identify patterns or inconsistencies across studies.
• Consultation with experts : Consulting with experts in the field can provide valuable insights into the quality and relevance of a study. Experts can also help identify potential limitations or biases in the study.
• Review of funding sources: Examining the funding sources of a study can help identify any potential conflicts of interest or biases that may have influenced the study design or interpretation of results.

Example of Evaluating Research

Example of Evaluating Research sample for students:

Title of the Study: The Effects of Social Media Use on Mental Health among College Students

Sample Size: 500 college students

Sampling Technique : Convenience sampling

• Sample Size: The sample size of 500 college students is a moderate sample size, which could be considered representative of the college student population. However, it would be more representative if the sample size was larger, or if a random sampling technique was used.
• Sampling Technique : Convenience sampling is a non-probability sampling technique, which means that the sample may not be representative of the population. This technique may introduce bias into the study since the participants are self-selected and may not be representative of the entire college student population. Therefore, the results of this study may not be generalizable to other populations.
• Participant Characteristics: The study does not provide any information about the demographic characteristics of the participants, such as age, gender, race, or socioeconomic status. This information is important because social media use and mental health may vary among different demographic groups.
• Data Collection Method: The study used a self-administered survey to collect data. Self-administered surveys may be subject to response bias and may not accurately reflect participants’ actual behaviors and experiences.
• Data Analysis: The study used descriptive statistics and regression analysis to analyze the data. Descriptive statistics provide a summary of the data, while regression analysis is used to examine the relationship between two or more variables. However, the study did not provide information about the statistical significance of the results or the effect sizes.

Overall, while the study provides some insights into the relationship between social media use and mental health among college students, the use of a convenience sampling technique and the lack of information about participant characteristics limit the generalizability of the findings. In addition, the use of self-administered surveys may introduce bias into the study, and the lack of information about the statistical significance of the results limits the interpretation of the findings.

Note*: Above mentioned example is just a sample for students. Do not copy and paste directly into your assignment. Kindly do your own research for academic purposes.

Applications of Evaluating Research

Here are some of the applications of evaluating research:

• Identifying reliable sources : By evaluating research, researchers, students, and other professionals can identify the most reliable sources of information to use in their work. They can determine the quality of research studies, including the methodology, sample size, data analysis, and conclusions.
• Validating findings: Evaluating research can help to validate findings from previous studies. By examining the methodology and results of a study, researchers can determine if the findings are reliable and if they can be used to inform future research.
• Identifying knowledge gaps: Evaluating research can also help to identify gaps in current knowledge. By examining the existing literature on a topic, researchers can determine areas where more research is needed, and they can design studies to address these gaps.
• Improving research quality : Evaluating research can help to improve the quality of future research. By examining the strengths and weaknesses of previous studies, researchers can design better studies and avoid common pitfalls.
• Informing policy and decision-making : Evaluating research is crucial in informing policy and decision-making in many fields. By examining the evidence base for a particular issue, policymakers can make informed decisions that are supported by the best available evidence.
• Enhancing education : Evaluating research is essential in enhancing education. Educators can use research findings to improve teaching methods, curriculum development, and student outcomes.

Purpose of Evaluating Research

Here are some of the key purposes of evaluating research:

• Determine the reliability and validity of research findings : By evaluating research, researchers can determine the quality of the study design, data collection, and analysis. They can determine whether the findings are reliable, valid, and generalizable to other populations.
• Identify the strengths and weaknesses of research studies: Evaluating research helps to identify the strengths and weaknesses of research studies, including potential biases, confounding factors, and limitations. This information can help researchers to design better studies in the future.
• Inform evidence-based decision-making: Evaluating research is crucial in informing evidence-based decision-making in many fields, including healthcare, education, and public policy. Policymakers, educators, and clinicians rely on research evidence to make informed decisions.
• Identify research gaps : By evaluating research, researchers can identify gaps in the existing literature and design studies to address these gaps. This process can help to advance knowledge and improve the quality of research in a particular field.
• Ensure research ethics and integrity : Evaluating research helps to ensure that research studies are conducted ethically and with integrity. Researchers must adhere to ethical guidelines to protect the welfare and rights of study participants and to maintain the trust of the public.

Characteristics Evaluating Research

Characteristics Evaluating Research are as follows:

• Research question/hypothesis: A good research question or hypothesis should be clear, concise, and well-defined. It should address a significant problem or issue in the field and be grounded in relevant theory or prior research.
• Study design: The research design should be appropriate for answering the research question and be clearly described in the study. The study design should also minimize bias and confounding variables.
• Sampling : The sample should be representative of the population of interest and the sampling method should be appropriate for the research question and study design.
• Data collection : The data collection methods should be reliable and valid, and the data should be accurately recorded and analyzed.
• Results : The results should be presented clearly and accurately, and the statistical analysis should be appropriate for the research question and study design.
• Interpretation of results : The interpretation of the results should be based on the data and not influenced by personal biases or preconceptions.
• Generalizability: The study findings should be generalizable to the population of interest and relevant to other settings or contexts.
• Contribution to the field : The study should make a significant contribution to the field and advance our understanding of the research question or issue.

Advantages of Evaluating Research

Evaluating research has several advantages, including:

• Ensuring accuracy and validity : By evaluating research, we can ensure that the research is accurate, valid, and reliable. This ensures that the findings are trustworthy and can be used to inform decision-making.
• Identifying gaps in knowledge : Evaluating research can help identify gaps in knowledge and areas where further research is needed. This can guide future research and help build a stronger evidence base.
• Promoting critical thinking: Evaluating research requires critical thinking skills, which can be applied in other areas of life. By evaluating research, individuals can develop their critical thinking skills and become more discerning consumers of information.
• Improving the quality of research : Evaluating research can help improve the quality of research by identifying areas where improvements can be made. This can lead to more rigorous research methods and better-quality research.
• Informing decision-making: By evaluating research, we can make informed decisions based on the evidence. This is particularly important in fields such as medicine and public health, where decisions can have significant consequences.
• Advancing the field : Evaluating research can help advance the field by identifying new research questions and areas of inquiry. This can lead to the development of new theories and the refinement of existing ones.

Limitations of Evaluating Research

Limitations of Evaluating Research are as follows:

• Time-consuming: Evaluating research can be time-consuming, particularly if the study is complex or requires specialized knowledge. This can be a barrier for individuals who are not experts in the field or who have limited time.
• Subjectivity : Evaluating research can be subjective, as different individuals may have different interpretations of the same study. This can lead to inconsistencies in the evaluation process and make it difficult to compare studies.
• Limited generalizability: The findings of a study may not be generalizable to other populations or contexts. This limits the usefulness of the study and may make it difficult to apply the findings to other settings.
• Publication bias: Research that does not find significant results may be less likely to be published, which can create a bias in the published literature. This can limit the amount of information available for evaluation.
• Lack of transparency: Some studies may not provide enough detail about their methods or results, making it difficult to evaluate their quality or validity.
• Funding bias : Research funded by particular organizations or industries may be biased towards the interests of the funder. This can influence the study design, methods, and interpretation of results.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Figures in Research Paper – Examples and Guide

Research Gap – Types, Examples and How to...

Research Paper Conclusion – Writing Guide and...

Informed Consent in Research – Types, Templates...

Research Topics – Ideas and Examples

Research Design – Types, Methods and Examples

We’re reviewing our resources this fall (September-December 2024). We will do our best to minimize disruption, but you might notice changes over the next few months as we correct errors & delete redundant resources. 

Critical Analysis and Evaluation

Many assignments ask you to   critique   and   evaluate   a source. Sources might include journal articles, books, websites, government documents, portfolios, podcasts, or presentations.

When you   critique,   you offer both negative and positive analysis of the content, writing, and structure of a source.

When   you   evaluate , you assess how successful a source is at presenting information, measured against a standard or certain criteria.

Elements of a critical analysis:

opinion + evidence from the article + justification

Your   opinion   is your thoughtful reaction to the piece.

Evidence from the article  offers some proof to back up your opinion.

The   justification   is an explanation of how you arrived at your opinion or why you think it’s true.

How do you critique and evaluate?

When critiquing and evaluating someone else’s writing/research, your purpose is to reach an   informed opinion   about a source. In order to do that, try these three steps:

• How do you feel?
• What surprised you?
• What left you confused?
• What pleased or annoyed you?
• What was interesting?
• What is the purpose of this text?
• Who is the intended audience?
• What kind of bias is there?
• What was missing?
• See our resource on analysis and synthesis ( Move From Research to Writing: How to Think ) for other examples of questions to ask.
• sophisticated
• interesting
• undocumented
• disorganized
• superficial
• unconventional
• inappropriate interpretation of evidence
• unsound or discredited methodology
• traditional
• unsubstantiated
• unsupported
• well-researched
• easy to understand
• Opinion : This article’s assessment of the power balance in cities is   confusing.
• Evidence:   It first says that the power to shape policy is evenly distributed among citizens, local government, and business (Rajal, 232).
• Justification :  but then it goes on to focus almost exclusively on business. Next, in a much shorter section, it combines the idea of citizens and local government into a single point of evidence. This leaves the reader with the impression that the citizens have no voice at all. It is   not helpful   in trying to determine the role of the common voter in shaping public policy.  

Sample criteria for critical analysis

Sometimes the assignment will specify what criteria to use when critiquing and evaluating a source. If not, consider the following prompts to approach your analysis. Choose the questions that are most suitable for your source.

• What do you think about the quality of the research? Is it significant?
• Did the author answer the question they set out to? Did the author prove their thesis?
• Did you find contradictions to other things you know?
• What new insight or connections did the author make?
• How does this piece fit within the context of your course, or the larger body of research in the field?
• The structure of an article or book is often dictated by standards of the discipline or a theoretical model. Did the piece meet those standards?
• Did the piece meet the needs of the intended audience?
• Was the material presented in an organized and logical fashion?
• Is the argument cohesive and convincing? Is the reasoning sound? Is there enough evidence?
• Is it easy to read? Is it clear and easy to understand, even if the concepts are sophisticated?

Research Evaluation

• First Online: 23 June 2020

Cite this chapter

• Carlo Ghezzi 2  

1054 Accesses

1 Citations

• The original version of this chapter was revised. A correction to this chapter can be found at https://doi.org/10.1007/978-3-030-45157-8_7

This chapter is about research evaluation. Evaluation is quintessential to research. It is traditionally performed through qualitative expert judgement. The chapter presents the main evaluation activities in which researchers can be engaged. It also introduces the current efforts towards devising quantitative research evaluation based on bibliometric indicators and critically discusses their limitations, along with their possible (limited and careful) use.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

19 october 2021.

The original version of the chapter was inadvertently published with an error. The chapter has now been corrected.

Notice that the taxonomy presented in Box 5.1 does not cover all kinds of scientific papers. As an example, it does not cover survey papers, which normally are not submitted to a conference.

Private institutions and industry may follow different schemes.

Adler, R., Ewing, J., Taylor, P.: Citation statistics: A report from the international mathematical union (imu) in cooperation with the international council of industrial and applied mathematics (iciam) and the institute of mathematical statistics (ims). Statistical Science 24 (1), 1–14 (2009). URL http://www.jstor.org/stable/20697661

Esposito, F., Ghezzi, C., Hermenegildo, M., Kirchner, H., Ong, L.: Informatics Research Evaluation. Informatics Europe (2018). URL https://www.informatics-europe.org/publications.html

Friedman, B., Schneider, F.B.: Incentivizing quality and impact: Evaluating scholarship in hiring, tenure, and promotion. Computing Research Association (2016). URL https://cra.org/resources/best-practice-memos/incentivizing-quality-and-impact-evaluating-scholarship-in-hiring-tenure-and-promotion/

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., Rafols, I.: Bibliometrics: The leiden manifesto for research metrics. Nature News 520 (7548), 429 (2015). https://doi.org/10.1038/520429a . URL http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351

Parnas, D.L.: Stop the numbers game. Commun. ACM 50 (11), 19–21 (2007). https://doi.org/10.1145/1297797.1297815 . URL http://doi.acm.org/10.1145/1297797.1297815

Patterson, D., Snyder, L., Ullman, J.: Evaluating computer scientists and engineers for promotion and tenure. Computing Research Association (1999). URL https://cra.org/resources/best-practice-memos/incentivizing-quality-and-impact-evaluating-scholarship-in-hiring-tenure-and-promotion/

Saenen, B., Borrell-Damian, L.: Reflections on University Research Assessment: key concepts, issues and actors. European University Association (2019). URL https://eua.eu/component/attachments/attachments.html?id=2144

Download references

Author information

Authors and affiliations.

Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy

Carlo Ghezzi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Carlo Ghezzi .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Ghezzi, C. (2020). Research Evaluation. In: Being a Researcher. Springer, Cham. https://doi.org/10.1007/978-3-030-45157-8_5

Download citation

DOI : https://doi.org/10.1007/978-3-030-45157-8_5

Published : 23 June 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-45156-1

Online ISBN : 978-3-030-45157-8

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Evaluation Research: Definition, Methods and Examples

Content Index

• What is evaluation research
• Why do evaluation research

Quantitative methods

Qualitative methods.

• Process evaluation research question examples
• Outcome evaluation research question examples

What is evaluation research?

Evaluation research, also known as program evaluation, refers to research purpose instead of a specific method. Evaluation research is the systematic assessment of the worth or merit of time, money, effort and resources spent in order to achieve a goal.

Evaluation research is closely related to but slightly different from more conventional social research . It uses many of the same methods used in traditional social research, but because it takes place within an organizational context, it requires team skills, interpersonal skills, management skills, political smartness, and other research skills that social research does not need much. Evaluation research also requires one to keep in mind the interests of the stakeholders.

Evaluation research is a type of applied research, and so it is intended to have some real-world effect.  Many methods like surveys and experiments can be used to do evaluation research. The process of evaluation research consisting of data analysis and reporting is a rigorous, systematic process that involves collecting data about organizations, processes, projects, services, and/or resources. Evaluation research enhances knowledge and decision-making, and leads to practical applications.

LEARN ABOUT: Action Research

Why do evaluation research?

The common goal of most evaluations is to extract meaningful information from the audience and provide valuable insights to evaluators such as sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. Most often, feedback is perceived value as useful if it helps in decision-making. However, evaluation research does not always create an impact that can be applied anywhere else, sometimes they fail to influence short-term decisions. It is also equally true that initially, it might seem to not have any influence, but can have a delayed impact when the situation is more favorable. In spite of this, there is a general agreement that the major goal of evaluation research should be to improve decision-making through the systematic utilization of measurable feedback.

Below are some of the benefits of evaluation research

• Gain insights about a project or program and its operations

Evaluation Research lets you understand what works and what doesn’t, where we were, where we are and where we are headed towards. You can find out the areas of improvement and identify strengths. So, it will help you to figure out what do you need to focus more on and if there are any threats to your business. You can also find out if there are currently hidden sectors in the market that are yet untapped.

• Improve practice

It is essential to gauge your past performance and understand what went wrong in order to deliver better services to your customers. Unless it is a two-way communication, there is no way to improve on what you have to offer. Evaluation research gives an opportunity to your employees and customers to express how they feel and if there’s anything they would like to change. It also lets you modify or adopt a practice such that it increases the chances of success.

• Assess the effects

After evaluating the efforts, you can see how well you are meeting objectives and targets. Evaluations let you measure if the intended benefits are really reaching the targeted audience and if yes, then how effectively.

• Build capacity

Evaluations help you to analyze the demand pattern and predict if you will need more funds, upgrade skills and improve the efficiency of operations. It lets you find the gaps in the production to delivery chain and possible ways to fill them.

Methods of evaluation research

All market research methods involve collecting and analyzing the data, making decisions about the validity of the information and deriving relevant inferences from it. Evaluation research comprises of planning, conducting and analyzing the results which include the use of data collection techniques and applying statistical methods.

Some of the evaluation methods which are quite popular are input measurement, output or performance measurement, impact or outcomes assessment, quality assessment, process evaluation, benchmarking, standards, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. There are also a few types of evaluations that do not always result in a meaningful assessment such as descriptive studies, formative evaluations, and implementation analysis. Evaluation research is more about information-processing and feedback functions of evaluation.

These methods can be broadly classified as quantitative and qualitative methods.

The outcome of the quantitative research methods is an answer to the questions below and is used to measure anything tangible.

• Who was involved?
• What were the outcomes?
• What was the price?

The best way to collect quantitative data is through surveys , questionnaires , and polls . You can also create pre-tests and post-tests, review existing documents and databases or gather clinical data.

Surveys are used to gather opinions, feedback or ideas of your employees or customers and consist of various question types . They can be conducted by a person face-to-face or by telephone, by mail, or online. Online surveys do not require the intervention of any human and are far more efficient and practical. You can see the survey results on dashboard of research tools and dig deeper using filter criteria based on various factors such as age, gender, location, etc. You can also keep survey logic such as branching, quotas, chain survey, looping, etc in the survey questions and reduce the time to both create and respond to the donor survey . You can also generate a number of reports that involve statistical formulae and present data that can be readily absorbed in the meetings. To learn more about how research tool works and whether it is suitable for you, sign up for a free account now.

Create a free account!

Quantitative data measure the depth and breadth of an initiative, for instance, the number of people who participated in the non-profit event, the number of people who enrolled for a new course at the university. Quantitative data collected before and after a program can show its results and impact.

The accuracy of quantitative data to be used for evaluation research depends on how well the sample represents the population, the ease of analysis, and their consistency. Quantitative methods can fail if the questions are not framed correctly and not distributed to the right audience. Also, quantitative data do not provide an understanding of the context and may not be apt for complex issues.

Learn more: Quantitative Market Research: The Complete Guide

Qualitative research methods are used where quantitative methods cannot solve the research problem , i.e. they are used to measure intangible values. They answer questions such as

• What is the value added?
• How satisfied are you with our service?
• How likely are you to recommend us to your friends?
• What will improve your experience?

LEARN ABOUT: Qualitative Interview

Qualitative data is collected through observation, interviews, case studies, and focus groups. The steps for creating a qualitative study involve examining, comparing and contrasting, and understanding patterns. Analysts conclude after identification of themes, clustering similar data, and finally reducing to points that make sense.

Observations may help explain behaviors as well as the social context that is generally not discovered by quantitative methods. Observations of behavior and body language can be done by watching a participant, recording audio or video. Structured interviews can be conducted with people alone or in a group under controlled conditions, or they may be asked open-ended qualitative research questions . Qualitative research methods are also used to understand a person’s perceptions and motivations.

LEARN ABOUT:  Social Communication Questionnaire

The strength of this method is that group discussion can provide ideas and stimulate memories with topics cascading as discussion occurs. The accuracy of qualitative data depends on how well contextual data explains complex issues and complements quantitative data. It helps get the answer of “why” and “how”, after getting an answer to “what”. The limitations of qualitative data for evaluation research are that they are subjective, time-consuming, costly and difficult to analyze and interpret.

Learn more: Qualitative Market Research: The Complete Guide

Survey software can be used for both the evaluation research methods. You can use above sample questions for evaluation research and send a survey in minutes using research software. Using a tool for research simplifies the process right from creating a survey, importing contacts, distributing the survey and generating reports that aid in research.

Examples of evaluation research

Evaluation research questions lay the foundation of a successful evaluation. They define the topics that will be evaluated. Keeping evaluation questions ready not only saves time and money, but also makes it easier to decide what data to collect, how to analyze it, and how to report it.

Evaluation research questions must be developed and agreed on in the planning stage, however, ready-made research templates can also be used.

Process evaluation research question examples:

• How often do you use our product in a day?
• Were approvals taken from all stakeholders?
• Can you report the issue from the system?
• Can you submit the feedback from the system?
• Was each task done as per the standard operating procedure?
• What were the barriers to the implementation of each task?
• Were any improvement areas discovered?

Outcome evaluation research question examples:

• How satisfied are you with our product?
• Did the program produce intended outcomes?
• What were the unintended outcomes?
• Has the program increased the knowledge of participants?
• Were the participants of the program employable before the course started?
• Do participants of the program have the skills to find a job after the course ended?
• Is the knowledge of participants better compared to those who did not participate in the program?

MORE LIKE THIS

Mass Personalization is not Personalization! — Tuesday CX Thoughts

Sep 24, 2024

Change Management Questions: How to Design & Ask Questions

Sep 23, 2024

Top 5 Change Management Models to Transform Your Organization

Sep 20, 2024

Customer Reviews: How to Ask and Easy Ways to Get Them

Sep 19, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

• Search Menu
• Sign in through your institution
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, 2. approach, 3. findings, 4. discussion, 5. conclusion, acknowledgements.

• < Previous

Changing research on research evaluation: A critical literature review to revisit the agenda

• Article contents
• Figures & tables
• Supplementary Data

Duncan A Thomas, Maria Nedeva, Mayra M Tirado, Merle Jacob, Changing research on research evaluation: A critical literature review to revisit the agenda, Research Evaluation , Volume 29, Issue 3, July 2020, Pages 275–288, https://doi.org/10.1093/reseval/rvaa008

• Permissions Icon Permissions

The current range and volume of research evaluation-related literature is extensive and incorporates scholarly and policy/practice-related perspectives. This reflects academic and practical interest over many decades and trails the changing funding and reputational modalities for universities, namely increased selectivity applied to institutional research funding streams and the perceived importance of university rankings and other reputational devices. To make sense of this highly diverse body of literature, we undertake a critical review of over 350 works constituting, in our view, the ‘state-of-the-art’ on institutional performance-based research evaluation arrangements (PREAs). We focus on PREAs because they are becoming the predominant means world-wide to allocate research funds and accrue reputation for universities. We highlight the themes addressed in the literature and offer critical commentary on the balance of scholarly and policy/practice-related orientations. We then reflect on five limitations to the state-of-the-art and propose a new agenda, and a change of perspective, to progress this area of research in future studies.

In this article, we undertake a critical review of over 350 relevant publications that together constitute, in our view, a diverse and wide-ranging literature ‘state-of-the-art’ on the performance-based research evaluation arrangements (PREAs) of universities and other public research organizations. These arrangements 1 address systematic evaluation exercises aiming to introduce resource and reputational policy incentives aligned with dominant notions of research quality (Langfeldt et al. 2019). We believe our analysis is necessary to: (1) highlight major themes addressed by literature; (2) provide a critical commentary on the balance of scholarly and policy/practice-related orientations in this literature; (3) identify limitations in this state-of-the-art; and finally (4) propose a novel research agenda to overcome these limitations.

Evaluations of policy and funding arrangements to support public research have been undertaken and studied for many decades. However, the number of studies on the details and effects of specific research evaluation arrangements globally increased considerably during the 1990s. This growing interest trails changing funding modalities for universities and public research organizations, with a rise of competitive, project grant funding, increased selectivity applied to institutional research funding streams ( Paradeise and Thoenig 2015 ), and the perceived importance of global rankings. Once pioneering research evaluation arrangements to allocate institutional funding, like Excellence for Research in Australia (ERA) and the UK Research Excellence Framework (REF), have also become established and seemingly intrusive enough to spur academic and policy concerns. This class of arrangements is becoming the predominant evaluative means to allocate public research funds and/or garner global reputation. It is, therefore, our central focus.

These PREAs have been discussed in an increasingly large body of both academic and grey literature sources, addressed via both scholarly and more policy/practice-related orientations. The scope of this literature varies widely. There are small-scale studies on peer judgement and dynamics of peer review panels operating inside broader national PREAs. There is also research on wider effects for behaviours and strategies of actors, organizations and institutions in national policy, and funding ‘research spaces’, for example, for universities, funding agencies, and researcher career trajectories (see Nedeva 2013 ; see also Smith, Ward and House 2011 ; Waitere et al. 2011 ; Lee, Pham and Gu 2013 ; Aagaard, Bloch and Schneider 2015 ; Reale et al. 2018 ; Whitley, Glaser and Laudel 2018 ; Lind 2019 ).

Research attention has been paid to increasing selectivity, and increasing use of performance-based allocation approaches in institutional research funding in countries like Australia, the Netherlands, Sweden, and the UK ( Organisation for Economic Co-operation and Development (OECD) 2009 ; Auranen and Nieminen 2010 ; Otley 2010 ; Wang and Hicks 2012 ; Tahar and Boutellier 2013 ; Leisyte and Westerheijden 2014 ; De Boer et al. 2015 ; Greenhalgh and Fahy 2015 ; Jonkers and Zacharewicz 2015 ; Arocena, Göransson and Sutz 2018 ; Canibano et al. 2018 ; Jonkers and Sachwald 2018 ; Woelert and McKenzie 2018 ). PREAs have also become of central importance in terms of research/epistemic governance. There is a perceived transition away from determination of research goals and orientations endogenously within universities and knowledge communities towards greater authority and influence from more strategic and managerial policy and university actors designing, deploying, or reacting to outcomes of PREAs ( Whitley and Gläser 2007 ; Langfeldt et al 2019).

Our critical review and analysis of this literature aims to identify thematic coverage, highlight limitations, and propose a new research agenda we believe is needed to move studies forward in this area. There have been previous surveys of research evaluation-related literature, for example, cross-sectional surveys and thematic reviews of evaluation practices and indicators (see De Rijcke et al. 2016 ). There have also been comprehensive studies correlating specific characteristics of differing national research evaluation arrangements to apparent national science system performance or excellence in international context (see Sandström and Van den Besselaar 2018 ; also Jonkers and Sachwald 2018 ). Whilst remaining within the confines of a critical review approach, our intent here is different and somewhat closer to meta-research motivations (c.f. Ioannidis 2018 ). We aim to analyse the themes, orientations, and limitations of research evaluation research itself, a review approach we believe has been overlooked in literature in this area to date.

In doing so we grapple with a messy reality. PREAs are dynamic, often politicized and are not ‘scientific’, static, standardized, or universal. They operate across multiple spatial levels and time horizons, use differing methods, involve varying degrees of transparency and costs, and are conducted by different kinds of organizations for various purposes ( Galleron et al. 2017 ). They can be understood as socially constructed systems, their legitimacy and effectiveness can be disputed, and they blend multifaceted contextual, political, managerial, economic, and reputational elements ( Bianco, Gras and Sutz 2016 ). We believe our critical review and analysis must therefore be purposive rather than trying to encompass all possible research on this vast topic.

To structure our article, first, we define our understanding of PREAs and use it to guide our approach. We describe our purpose in collecting and coding a bespoke dataset of 354 pieces of literature that we believe constitutes the most relevant ‘state-of-the-art’ on PREAs. Second, we present an analysis of five research themes we derive through inductive clustering of this state-of-the-art and provide critical commentary on the major arguments in this body of research. Third, we discuss five limitations to this PREA-related literature and suggest a novel research agenda to address them.

We understand PREAs as including ‘organized sets of procedures for assessing the merits of research undertaken in publicly funded organizations that are implemented on a regular basis, usually by state or state-delegated agencies’ ( Whitley and Gläser 2007 : 6). PREAs operate at multiple levels, as an ‘ensemble of practices and institutional arrangements in a country’ and/or locally in a university organization, mediating ‘between scientific quality controls and research policies’ ( Cruz-Castro and Sanz-Menéndez 2007 : 205). They are part of the ‘organizational governance’ of universities, in directing ‘strategy, funding’ and operations, and are a potential source of tensions ( Luo, Ordóñez-Matamoros and Kuhlmann 2019 : 1). They are also frequently ‘intended to change science by improving its quality’ and possibly even altering research ‘content’ ( Gläser 2007 : 245). They can be ‘weak’ and aim primarily at ‘information-gathering’ for benchmarking of research, researchers, and research organizations—or else ‘strong’ in performance-based ‘national systems of research output evaluation’ and be used as a basis ‘to distribute research funding to universities’ ( Hicks 2012 : 260).

To guide our critical review and analysis of the literature, we capture the most salient of these aspects by defining PREAs here as the institutionalized, or semi-institutionalized, practices and procedures aiming to assess the merit of the research output, research environment, and research engagement of research organizations with a view to incentivizing desired change or continued performance . PREAs may be conducted at different levels of social aggregation—for example, national research system, organization, etc.—and affect resource allocation and reputations.

Not to conflate our definition of PREAs with other possible forms of evaluation, we draw upon an understanding of science dynamics as involving research fields and research spaces ( Nedeva 2013 ). We thus distinguish between PREAs and two other commonly addressed types of research evaluation. Our critical review includes only literature on PREAs located in the research space (see Figure 1 ). We thereby exclude literature addressing research evaluation types performed by research organizations and research field-related knowledge claim assessment.

Schematic of our literature critical review search strategy to include PREAs and exclude other evaluation types (adapted from Nedeva 2013 ).

This PREA definition directed our focus to the field of science, technology, and innovation policy (STIP) studies, and mapped onto central and peripheral journals in this area. Our approach followed that of a critical narrative review ; we wished to identify key contributions around our specified topic but not necessarily to address all evaluation-related material ever produced (c.f. Demiris, Oliver and Washington 2019 ). Our definition directed us to core STIP-related journals (e.g. Research Evaluation , Research Policy , Science and Public Policy , Scientometrics , and Minerva ) and selected peripheral ones. 2

We used the keywords ‘research evaluation’, ‘institutional’, and ‘university(ies)’ in searches of (1) Web of Science, (2) Scopus, and (3) Google Scholar. This resulted in 675 hits from numerous journals, books, and non-academic sources. We reviewed titles and abstracts at this stage to screen for duplicates and, guided by our PREA definition, ensured materials primarily addressed research space-related research evaluation. This was done using (1) our knowledge as active scholars in fields of research evaluation and research policy for several decades (c.f. Adler and Adler 1987 ); (2) our knowledge of research consultancies and their key reports; and (3) invited expert advice by email, telephone, and face-to-face from a small number of international research policy/evaluation academic and consultant colleagues (this latter element introduced an element of consensus narrative review; c.f. Wilczynski 2017 ). Our final set thus also included grey literature from consultancies and funders like Technopolis, PA Consulting, the European Commission, the former Higher Education Funding Council for England (HEFCE), and select others. 3

This critical narrative review process with an element of consensus review led to our final set of 354 full-text materials, including academic articles, books, funder and policy reports that we then inductively coded and analysed. The earliest piece of literature that we retrieved was published in 1968. For convenience, we set 2018 as a cut-off publication year. Just over 85% of the literature we included and reviewed in this bespoke dataset was published between 2000 and 2015—reflecting increased attention as funding modalities and evaluation arrangements have been recently changing. A total of 179 items were primarily qualitative, 103 were quantitative, and 72 were mixed methods based. The literature in the dataset addressed PREAs related to 37 countries and territories, trans-national arrangements, and international surveys of these arrangements (e.g. by the European Union [EU] and the Organisation for Economic Co-operation and Development [OECD]). 4

Following this highly selective, expert-informed, critical and consensus narrative review approach we cannot claim to have produced a comprehensive collection of all materials ever published on ‘research evaluation’-related topics. However, we believe we captured enough breadth and depth of the ‘state-of-the-art’ on PREA-related topics to satisfy our purposive analysis, to highlight key limitations, and to underpin our proposition of a novel research agenda.

For every piece of literature in the dataset we manually read abstracts and full texts. From this reading, we wrote synopses summarizing the approach, coverage, findings, and conclusions of each piece of literature. We then analysed our database of synopses to produce an inductive clustering of all the literature into five major themes , shown in Table 1 . All literature was assigned to a single major theme based upon primary message. This was based on our subjective reading of the literature content, what proportion of it addressed a given theme, and the prominence afforded that theme in the literature. 5

Overview of the five major themes we produced to cluster our database, and further details of the 354 pieces of PREA-related literature

Our first inductive clustering theme, accounts of local PREAs was where we assigned literature whose primary content provided ‘thick descriptions’. This included case studies of PREAs specific to a national research system (e.g. ERA or REF), a trans-national regional bloc (e.g. EU-level arrangements), for a sub-national region, for a specific organization (e.g. university), or for a sector or grouping of organizations (e.g. medical research in universities and research institutes). Our second theme was where we clustered comparative studies of PREAs , for instance, those comparing specific sets of countries or specific research fields. Our third theme captured literature providing discussions of rationales for (performance-based) research evaluation , for example, discussing the policy impetus for performance-based criteria and how they related to pursuit of excellence aims, efficiency, and other concerns. The fourth theme clustered appraisals of (performance-based) research evaluation methodologies ; for example, debates around the relative merits of bibliometrics, altmetrics, and other indicators vis-à-vis peer review practices—essentially the detailed methods and machinery, technical parameters, and logistics of the design and deployment of PREAs. Our fifth and final theme clustered literature attempting studies of effects on the science system , for example, how PREAs interacted with science dynamics and researcher careers.

We found it helpful to characterize the literature further using limited additional coding: literature type (i.e. journal articles, books or book chapters, policy reports); literature content —primary research (e.g. interviews, surveys, bibliometrics, mathematical models and simulations, mixed methods) or secondary (e.g. desk-based literature reviews and/or secondary sources); literature methods (quantitative, qualitative or mixed); literature approach , that is, thick descriptions of specific cases, critical analyses, and attempts at comparative analysis; and object of analysis, that is, organization level evaluations or sub-national, national, or trans-national levels. 6 These further codes are shown in Table 2 and were included in our analytical approach. 7

Description of our PREA-related literature database coding

Inductively clustering these five themes and using our further coding we began our purposive analysis, where we posed five specific questions:

What key themes have been addressed by this literature?

What is the balance of research attention across all the themes?

What are the analytical implications of the apparent balance between scholarly and policy/practice-orientations in this literature?

What aspects have not been addressed?

Given this state-of-the-art, what new research agenda might move PREA-related research forward?

We now present our analysis of the dataset of 354 pieces of literature. For each of the five themes, we provide a summary of key research arguments, brief critical commentary, and descriptive information using our further codes. 8

3.1 Theme 1: Accounts of local PREAs

Theme 1 grouped literature we determined to be primarily focused on providing descriptive accounts of local PREAs . Altogether we assigned 100 pieces of literature to this theme. A total of 77 pieces described PREAs at national level, for example, national evaluations like those in Australia, the Netherlands, and the UK. Within this theme, we also placed literature primarily describing arrangements at organizational (six pieces of literature), sub-national (five pieces), and trans-national levels (11 pieces). 9

National level PREAs were described for countries where these practices were already well established, like the UK ( Barker 2007 ; see also Martin and Whitley 2010 ; Morris 2010 ) and Australia ( Butler 2008 ; Donovan 2008 ). These arrangements were also described in other literature, to show them as apparent exemplars for development and implementation of new arrangements in countries or regions that had previously not used such practices ( Fiala 2013 ; Ancaiani et al. 2015 ; see also European Centre for Strategic Management of Universities (ESMU) 2010 ; Geuna and Piolatto 2016 ). Some literature included not only primary descriptive content but also secondary messages, such as critical feedback to policymakers, and reflections on existing arrangements for possible policy learning ( Hare 2003 ; Adams and Gurney 2010 ; Elsevier 2013 ; Hughes, Kitson and Bullock 2013 ; Van Drooge et al. 2013 ; Higher Education Funding Council for England (HEFCE) 2014 ; Wouters et al. 2015 ; see also Henkel 1999 ; Auranen and Nieminen 2010 ; Broadbent 2010 ; Elsevier 2011 ; Spaapen and Van Drooge 2011 ).

Literature we grouped in Theme 1 had often been commissioned by national or international organizations responsible for evaluating research outputs, environments and engagements of higher education institutions or other research organizations. Nearly half the material in Theme 1 (46 pieces of literature) was policy reports describing national-level arrangements, then benchmarking them against each other to provide an international overview. These kinds of policy reports were commissioned and (presumably) funded by ministries of education in different countries, the OECD and say, the former HEFCE in the UK. We determined these bodies had funded these studies to enable policy learning about past experiences and/or arrangements used in other countries.

The bulk of Theme 1 literature we would call ‘highly descriptive’ (81 pieces of literature). We determined they used no explicit theoretical positions. A similar number used primarily qualitative and/or mixed methodologies (81 pieces). Ten pieces of Theme 1 literature had what we would consider more analytical approaches; 19 used quantitative methodologies, for example, Cattaneo, Meoli and Signori (2016) (see also Frølich 2008 , 2011 ; Frølich, Schmidt and Rosa 2010 ; Wang and Hicks 2012 ; Frankel, Goddard and Ransow 2014 ; Hamann 2016 ); and 42 of the 100 pieces collected primary data. The others based their descriptive accounts on secondary research and sources.

The descriptions of PREAs across Theme 1 literature addressed the following: descriptions of national-level arrangements (broad and fine details); evaluation strategies (apparent purposes, economic and social rationales); funding mechanisms (i.e. whether and how much evaluation results were linked to funding streams); assessment methods and inclusion/exclusion criteria of what was assessed; how often assessment took place; what units were assessed (research themes, research organizations, etc.); and evaluation outcomes (e.g. apparent levels of research-related performance of organizations, regions or nations, based on indicators such as publication volumes, citations, number of patents, and/or university–industry links). Theme 1 literature primarily used case study research designs and detailed the—sometimes considerable—costs associated with (repeated) use of research evaluation. Some provided cost-benefit analyses of existing evaluation exercises (e.g. see Campbell and Boxall 2004 ; PA Consulting Group 2008 ; Technopolis 2009 , 2010 ; see also Mahieu, Arnold and Kolarz 2013 , 2014 ; Arnold et al. 2014 ; Mahieu and Arnold 2015 ). We classed these pieces of literature as largely ‘user-driven’. They seemed designed to answer research questions or address research interests of policymakers and evaluation practitioners.

Turning a critical eye to Theme 1 literature, we found an absence of frameworks for theoretically or conceptually based study and analysis of PREAs. Theme 1 literature was primarily descriptive —both for the material published in academic journals and ‘grey literature’, user-driven, policy reports. This potentially presents a problem and may not be an ideal basis to support robust policy learning. This literature in our critical opinion does not provide analysis and comprehension of social mechanisms around PREAs. However, it clearly does provide a source of rich empirical material and cases that could later be revisited for analytical purposes.

3.2 Theme 2: Comparative studies of PREAs

We assigned 40 pieces of literature into our clustering Theme 2. These were comparative studies of PREAs , comparing, for example, arrangements for specific sets of countries, or for particular research fields. Some undertook broad comparisons of institutional and other evaluation arrangements ( Geuna and Martin 2003 ; Orr 2004 ; Hicks 2010 ; Arnold and Mahieu 2015 ; see also Frølich 2008 ; Geuna and Piolatto 2016 ; Sandström and Van den Besselaar 2018 ). Others compared selective research funding arrangements, effects for behaviours like research collaboration ( Johnston 1994 ), actions of research funding agencies ( Lepori et al. 2009 ), consequences of evaluation for university funding ( Franzoni, Scellato and Stephan 2011 ; see also Sörlin 2007 ), or PREA-related criteria for assessing research quality in different fields ( Hug, Ochsner and Daniel 2013 ).

Literature here provided accounts of PREAs in multiple different settings and countries, but crucially with few attempts at analytical comparison. Hicks (2010) , for instance, compared specific research evaluation objectives and strategies used by EU countries, Australia, South Africa, and some Asian countries—but did not compare wholescale the design, operation, and effects of these arrangements within a comprehensive framework. Rebora and Turri (2013 ; see also Geuna and Piolatto 2016 ) compared how research funding of universities evolved over time to incorporate selectivity and evaluation elements, specifically in the UK and Italy. Similarly, Geuna and Martin (2003) compared specific methods of evaluation used in 12 countries in Europe and the Asia-Pacific region.

Like Theme 1, the majority of Theme 2 literature we considered user-driven policy reports (26 pieces of literature or 65% of this theme was policy reports; 14 were academic publications, i.e. journal articles, a book, and a book chapter). Some Theme 2 literature also compared PREA-related practices across different countries to support policy learning ( Iorwerth 2005 ; Grant 2010 ) or as guidance for policymakers wishing to implement and institutionalize PREAs in new settings (see e.g. Arnold and Mahieu 2015 ). Theme 2 literature was largely based on secondary research (in 29 pieces or 73% of Theme 2) and used qualitative or mixed research methods (88% of literature in Theme 2).

3.3 Theme 3: Discussions of rationales for (performance-based) research evaluation

Literature we clustered into Theme 3 primarily provided discussions of rationales for research evaluation , for example, the policy impetus and rationales for using performance-based evaluation criteria or how policy concerns and performance criteria like excellence and efficiency were interrelated. This Theme 3 was a very specific sub-set of the literature. It was our smallest cluster, at only 18 pieces. 10 Some analytical frameworks were present in Theme 3 but no common or shared framework was used across different literature here. 11

A first key argument in the Theme 3 literature was that the introduction of PREAs requires that one also consider value-for-money and issues of research quality. Here, Theme 3 literature suggested policymakers’ rationales included values like promoting knowledge-based economies and strongly overlapped with efforts to use public research systems in different national settings to revive and/or restructure the orientation and/or performance of whole national economies (e.g. Rip and van der Meulen 1996 ; Bozeman and Sarewitz 2011 ; Sørensen, Bloch and Young 2015 ; Jonkers and Sachwald 2018 ; see also Elton 2000 ; Geuna and Martin 2003 ; Bence and Oppenheim 2005 ; Frølich, Schmidt and Rosa 2010 ; Martin and Whitley 2010 ; Mingers and White 2015 ; Woelert 2015 ).

A second key argument was that PREAs are evolving in parallel with rationales asserting that more competitive allocation of research funding improves research performance, for example, as judged by measures like publication productivity, and other indicators of apparent ‘excellence’. Theme 3 therefore seemed to include an emerging, critical research tradition moving close to addressing effects of competitive funding interventions as part of evolving PREAs. The interweaving of competitive funding and research evaluation was treated from research funders’ perspectives, at national research system level and in some cases at the level of researchers ( Benner and Sandström 2000 ; Smith, Ward and House 2011 ; see also Sørensen, Bloch and Young 2015 ).

Theme 3 literature suggested consideration of PREAs has to account for public research funding becoming more fine-grained over time. Previous research funding regimes generally treated most if not all aspects of the research system like a 'black box'. For instance, literature here described ‘first generation’ institutional research funding streams that did not address researchers, but simply took universities in the aggregate. Later approaches channelled funding streams by disaggregating research systems into actors, institutions, etc. and attempted to leverage specific types of outcomes or promote specific behaviours. Theme 3 literature documented and critiqued this shift, noting the journey of the word ‘excellence’, in particular, from being an idiosyncratic, field-specific term used by researchers to recognize extraordinary scientific contributions, to it becoming an indicator used by policymakers and university managers to refer to all research as sharing some predefined set of characteristics (e.g. Laudel 2005 ; Smith, Ward and House 2011 ; see also Hicks 2012 ; Sousa and Brennan 2014 ). For example, Sørensen, Bloch and Young (2015) concluded that when ‘excellence’ was discussed in the context of PREAs it had now moved from being a marker of purely scientific performance to a broader basket of additional research performance-related criteria, for example, potential commercialization of research outputs, and indeed anything ‘commercializable’.

A third key argument in Theme 3 literature was a travel of global policy and economic competitiveness discourse into PREAs. The rise and diffusion of ideas (and ideology) around the global competition for knowledge, resource constraints, and resultant changing views of universities were chronicled, that is, a change from them being civic, public organizations to being more like corporations, and venues where performance must be audited. Theme 3 literature considered how conceptions of knowledge have shifted, and excellence has become a means within PREAs to reward ‘winners’ and punish ‘losers’. This was described as a new ‘strategic approach’ to research policy and resource allocation through these arrangements, suggesting policymakers and governments have moved closer, in theory if not yet in practice, to selecting and affecting the types and topics of research, research content (methodologies, equipment), and even which specific researchers they believe can deliver ‘excellence’ within a particular research system ( Benner and Sandström 2000 ; Sörlin 2007 ; see also Hicks 2012 ; Watermeyer 2014 , 2016 ). 12 An apparent merging was noted, of policymakers’ search for ‘excellence’ and use of evaluation as a tool to measure research system effectiveness, with guiding and directing socio-economic investment decisions.

All bar one piece of literature in Theme 3 was published in academic journals. Theme 3 literature drew mainly on secondary data, used qualitative methods, and was the most analytical set, in our view. Literature here attempted to unpack varying, evolving rationales for PREAs, and to trace how they were now being seen as enablers of structural change, and as facilitating national systems that could compete more at an international level.

3.4 Theme 4: Appraisals of (performance-based) research evaluation methodologies

Nearly a third of all the pieces of literature in our database (103 pieces, 29% of the full dataset) addressed methods related to PREAs, for example, whether and which indicators were reliable measures or proxies to evaluate research performance, in terms of excellence and quality ( Cozzens 1981 ; Donovan 2007 ; De Jong et al. 2011 ; Wunsch-Vincent 2012 ; Wilsdon et al. 2015 ; see also Aagaard 2015 ). These pieces we clustered in Theme 4. Literature here we judged as aiming to discover or design the ‘best’ methods for PREAs to assess subjective notions like research excellence and quality. Some favoured exclusive use of peer review or of bibliometrics. Others advocated mixed approaches say, combining peer review and bibliometrics techniques ( Butler 2007 ; Abramo, D’Angelo and Di Costa 2008 , 2011 ; Abramo and D’Angelo 2011 ; Franceschet and Costantini 2011 ; Abramo, Cicero and D’Angelo 2013 ; Eyre-Walker and Stoletzki 2013 ).

Theme 4 literature was very useful in highlighting two current dilemmas around design and deployment of differing PREAs. First, materials here considered which approach should be used, that is, predominantly qualitative or quantitative? Some literature addressed whether qualitative peer review was the most appropriate and/or cost-effective instrument to use or whether use of bibliometrics and other kinds of quantitative indicators was preferable. Other literature advocated use of blended or mixed approaches. Bertocchi et al. (2015) , for instance, suggested research performance be evaluated using bibliometrics as an initial input for subsequent peer review. Still others proposed bibliometrics be used at national or local level to manage and/or monitor research performance within an evaluation, before feeding into later large-scale, peer review-based judgements, that is, so-called ‘informed’ peer review (see Neufeld and von Ins 2011 ; Wilsdon et al. 2015 ).

A second dilemma in Theme 4 literature was how current methodologies might be modified for use by policymakers and/or university managers to encourage , or at least not impede sustainable research activity in specific fields (e.g. in social sciences and humanities, SSH) or to foster research with particular properties (e.g. breakthrough, frontier, long-term). For instance, in SSH ‘informed’ peer review was advocated to assess better the performance of research fields where publishing journal articles represent only part of research outputs activities (e.g. in political science, where books and policy engagement also occur, Donovan 2009 ). Other literature suggested the same approach be part of PREAs in fields where peer review was dominated by reviewers representing only specific sub-fields (e.g. all denominations of economists being evaluated only by neoclassical/mainstream economists; Lee and Harley 1998 ; Lee, Pham and Gu 2013 ). Theme 4 literature advocated or designed new field-specific, more ‘inclusive’ quantitative indicators (e.g. social media-related ‘altmetrics’) to account for societal effects, broader or ‘alternative’ research outputs, interactions, exchanges, and outcomes ( Bozeman, Dietz and Gaughan 2001 ; Kaufmann and Kasztler 2009 ; Kenna and Berche 2011 ; Ochsner, Hug and Daniel 2012 ; Kwok 2013 ; Sastry and Bekhradnia 2014 ).

Theme 4 literature predominantly featured material published in academic journals (83 of the pieces or 80% of Theme 4), relied on secondary data (80 pieces) and used quantitative methodologies (58 pieces). The predominant object of analysis was PREAs at national level (in 63 pieces of literature).

3.5 Theme 5: Studies of effects on the science system

Our final Theme 5 covered studies of effects on the science system from PREAs. Here, we clustered 93 pieces of literature, addressing effects at multiple spatial levels (regional, national, trans-national) and analytical levels (system, organization, researcher, research topics and content). Some literature instead took a cross-cutting view across these levels. Effects of PREAs on universities specifically were a dominant focus. Other literature combined this with attention to a general shift away from institutional/block funding towards proportionally more of competitive, project-based research funding allocation. Few pieces of literature addressed effects of PREAs upon additional parts of the science system beyond universities, say, effects for global research fields or aggregate effects at global level of multiple differing arrangements operating in parallel at national and/or regional levels.

Some Theme 5 literature argued specific PREAs have generated effects at the ‘macro’ level of changing how science, universities, and scientists/researchers are perceived by society. The critical view was that strategic use by policymakers and university managers of particular arrangements—with perhaps disproportionate emphasis here upon the UK’s Research Assessment Exercises (RAEs) and REF—had significantly changed organizational conditions for, and authority relations around knowledge creation ( Himanen et al. 2009 ; see also De Jong et al. 2011 ; Kallerud et al. 2011 ; Whitley, Glaser and Laudel 2018 ).

At ‘meso’ level, literature observed that publicly funded research universities had become vulnerable to, and at risk of, being transformed by what certain exogenous stakeholders (e.g. politicians, policymakers, research funding agencies, corporate actors) considered ‘best’ for them. They were portrayed as losing autonomy, scholarly leadership, and ability to generate new and/or critical academic ideas. Universities and their researchers were framed as forced to abandon Mertonian notions of autonomy, disciplinarity, and freedom (c.f. Merton 1968 ) and expected to adopt values and quality standards shaped by outside demands ( Frølich, Schmidt and Rosa 2010 ; Harland et al. 2010 ; see also Luukkonen 1997 ; Van der Meulen 1998 ; Ferlie, Musselin and Andresani 2008 ). Universities were diagnosed as no longer doing what they were ‘best’ at, and as complying with exogenous quality and excellence standards imposed by PREAs—or forced to suffer consequences of reduced research revenue and/or national and global reputation in local and world rankings/league tables ( Knowles and Burrows 2014 ; see also Elton 2000 ; Luukkonen and Thomas 2016 ).

Other effects on universities included university management practices described as moving away from traditional ‘academic’ values ( Linkova 2014 ; Agyemang and Broadbent 2015 ), changed university hiring, probation, and promotion strategies, allied to university strategic objectives and management practices becoming strongly coupled to criteria derived from evaluation-related goals and targets (see also Henkel 1999 ). Universities were also framed as embracing competition rather than resisting it and using PREAs at ‘micro’ level, to develop and deploy incentives, and ever more granular research information systems, monitoring and auditing mechanisms, to foster, reward, or sanction particular kinds of research productivity by research groups and at individual researcher level ( Nedeva et al. 2012 ).

Other reported ‘meso’ level effects were university management game-playing, particularly within ‘strong’ PREAs directly linked to resource allocation ( Whitley, Glaser and Laudel 2018 ). Universities, their leaders, and managers were reported as developing and using deliberate strategies to incentivize and direct types of research, researchers, and external university-stakeholder relationships that painted them in the most favourable light within PREAs so as to maximize research funding capture (again, particularly relating to the UK’s RAEs/REF). This behaviour reportedly has led to: undesirable concentration of resources by funders and universities to support short-term ‘safe’ rather than long-term risky research; allocation of resources to meet lay stakeholder/proxy indicators of excellence irrespective of knowledge community/substantive judgements about research quality; favouring competition over collaboration, thus risking fragmentation of academic/professional collegiality and reciprocity within and across universities; and direct or indirect promotion of ‘salami slicing’ publication practices to reward publication of a greater quantity of perhaps less comprehensive research works rather than focus on fewer but potentially more significant publications of ‘higher’ quality ( Butler 2003 ; Leisyte and Westerheijden 2014 ; see also Abramo, D’Angelo and Di Costa 2011 ).

Further effects were reported to be: increased short-termism generally at universities; superficial attention to what in some quarters are seen as spurious markers of university reputation/excellence in national and global league tables for universities ‘playing the game’; erosion of creativity; reduced diversity of the research topics, methods and approaches researchers’ pursue; and strategy and management level distortions in resource allocations that undermine previous synergies between teaching and research ( Whitley, Glaser and Laudel 2018 ; see also Paradeise and Thoenig 2015 ). Some authors even felt ‘strong’ PREAs (i.e. coupled to funding allocation) and audit cultures ‘dehumanized’ researchers and harmed traditional, more liberal, long-standing purposes and roles of universities in wider society ( Hare 2003 ; Harland et al. 2010 ; Olssen 2016 ; see also Geuna and Martin 2003 ; Martin and Whitley 2010 ).

Some Theme 5 literature addressed effects at the ‘micro’ level of researchers and their research work processes: apparent loss of academic work-life balance and freedom; downgrading of teaching relative to research/publications; loss of intellectual curiosity; and a debasing of the general character of academic scholarship ( Court 1999 ; Roberts 2007 ; Linkova 2014 ; Vincent 2015 ). Reported centralization of authority towards organizational elites like university managers, using expanding research data systems and information sourced from national/external and local/internal PREAs, were considered avenues of (negative) control over research content ( Gläser et al. 2010 ; see also Aagaard 2015 ). PREAs were also reported to increase administrative burdens for researchers and decrease research time and productivity ( Martin 2016 , 2011 ).

Other Theme 5 literature indicated a fundamental transformation cutting across macro/meso/micro levels that had reportedly changed: university (research) culture; the nature, remit, processes, and practices of universities’ objectives and goals; the relevance of university research; and research topic coverage and diversity. These effects were linked to changing university strategies to mobilize the outcomes of PREAs to improve positioning in university rankings ( Martin 2011 ; Holmes 2015 ). Academia and knowledge were described as being reconceptualized as commodities, driven by economic efficiency and value-for-money concerns. A shift towards performativity was reported, with universities and academics assigned and/or adopting new purposes within these changing authority relations ( Harland et al. 2010 ; Whitley 2011 ). These relations included policymakers, and university managers, administrators, and field elites in universities using their newfound authority to attempt to ‘steer’ science systems even at the expense of marginalizing input from academics and other voices. Some authors here sounded a ‘wake-up call’ for academics to resist supposedly harmful use of PREAs and fight to retain long-held values that give meaning to ‘the academy’ ( Martin and Whitley 2010 ; Martin 2011 , 2016 ; Waitere et al. 2011 ; see also Bence and Oppenheim 2005 ; Murphy and Sage 2014 ). Authors contended PREAs should prove their usefulness in improving research culture, financial sustainability, research capacity, and so on in universities—rather than that academics should bow and bend to fit better the parameters of these arrangements. Some authors here foreshadowed an ‘end’ to universities as places for reflection and creative thinking, extinguished by the utilitarian influence of PREAs—even those PREAs that advocate and incentivize seemingly more positive societal ‘impact’ from research ( Knowles and Burrows 2014 ; see also Claeys-Kulik and Estermann 2015 ).

Other Theme 5 literature reported changes to the global communication system of science. Academic journal editors were reported as developing strategies to inflate their own journal rankings and citation counts to pander to use of PREAs and thus to become more attractive to authors ( Gibson, Anderson and Tressler 2014 ). Journal editors were criticized for apparently seeking fewer path-breaking, critical research ideas and methods to publish (that reportedly accumulate citations more slowly), instead favouring more immediately citable, fashionable topics and approaches that can quickly inflate journal impact factors. Some Theme 5 literature described academic editors, publishers, reviewers, universities, government, and funding agencies as collectively adapting here to PREAs ( Macdonald and Kam 2010 ; Watermeyer 2016 ).

We make two main critical points about this Theme 5 literature. First, little is known about causal relationships between PREAs and many if not all of these reported changes and apparent effects (see also Gläser 2019 ). This holds true for micro-level changes in research topic selection and researchers’ pursuit of research programmes/lines and for other levels ( Waitere et al. 2011 ; De Rijcke et al. 2016 ; Hammarfelt and de Rijcke 2015 ; see also Laudel 2005 ; Whitley and Gläser 2007 ). There are inherent methodological difficulties to measure and attribute PREA-related change here within and across heavily mediated, multi-level, multi-actor, regional, national, and trans-national research funding and policy ‘spaces’ and global ‘research fields’ ( Nedeva et al. 2012 ; Whitley, Glaser and Laudel 2018 ).

Second, this literature may be biased by over-representation of both scholarly and more personal accounts/normative responses to the UK RAEs/REF. The UK’s primary PREA is globally influential, but we must remember it is not necessarily ‘best practice’, has not travelled to many other regions of the world, and analytically the UK is an outlier or ‘unique’ ( Sivertsen 2017 ). Reported effects there cannot be taken to be representative of effects of differing arrangements in other contexts (this criticism of course also ties in with the lack of comparative analytical frameworks across the literature state-of-the-art). There are few attempts to distinguish analytically the RAEs/REF from other PREAs or to make theory-based assumptions and arguments to link causally particular arrangements to specific effects.

In overview, most Theme 5 literature was published in academic journals (82 pieces of literature or 88% of this theme). Many arguments were built on either primary (43 pieces) or secondary data (50 pieces) and used qualitative approaches (in 61 pieces of literature). We considered most Theme 5 literature to be predominantly analytical in approach (54 pieces).

3.6 Cross-cutting issues

Looking across all five clustering themes most literature seemed to share the view that, whatever the specific arrangements, PREAs are ‘here to stay’ (e.g. Martin and Whitley 2010 ; League of European Research Universities (LERU) 2012 ). There was resigned acceptance that although PREAs remain contentious, and evidence about their operation is uneven, they nevertheless are considered useful for multiple purposes. They enable governments to map, prioritize, and capitalize (better) upon research and researcher capacity within a science system. They are an accepted means to allocate research funding and infrastructure resources based upon such maps, prioritizations, and investment plans and strategies (e.g. Strehl, Reisinger and Kalatschan 2007 ; European Commission 2009 ; Hicks 2010 ; Olson and Rapporteurs 2011 ; Organisation for Economic Co-operation and Development (OECD) 2011 ; Cunningham, Salavetz and Tuytens 2012 ; Mahieu, Arnold and Kolarz 2013 ; Higher Education Funding Council for England (HEFCE) 2014 ; Arocena, Göransson and Sutz 2018 ).

Literature often neither sought nor found standardization or ‘best practice’ of PREAs. There remain open questions, and unresolved debates, for example, how to improve design and deployment of PREA-related strategies, research funding mechanisms, performance assessment methods, key criteria, how often to conduct evaluation, whether to evaluate academic and/or non-academic research, whether to distinguish between researchers and research environments, and how to determine the most appropriate unit(s) and subject(s) of assessment (e.g. Wooding and Grant 2003 ; Organisation for Economic Co-operation and Development (OECD) 2009 , 2010a , b ; Ministry of Education 2012 ; Reale et al. 2018 ; see also Sivertsen 2017 ; Regan and Henchion 2019 ).

Despite this agnosticism regarding ‘best’ arrangements, there were fears of isomorphism—particularly of widespread diffusion of the UK’s RAE/REF arrangements, either in entirety or specific elements, like arrangements to evaluate research ‘impact’. Patterns of exploration, testing, and learning by various stakeholders (e.g. research funders, policymakers) were seen as enabling such adoption, translation, travel, and/or transplantation of PREAs from one country, region, or university context to another. Similarly, pathways were observed for ‘trickle down’ of national arrangements into bespoke—and sometimes highly contentious—local arrangements inside particular universities and other public research organizations (e.g. Iorwerth 2005 ; Mahieu, Arnold and Kolarz 2013 ; Ohtani, Kamo and Kobayashi 2013 ; Mok 2014 ; Aagaard 2015 ; Geuna and Piolatto 2016 ; Woelert and McKenzie 2018 ; see also Lind 2019 ).

4.1 Limitations of literature on research evaluation arrangements?

Our analysis suggests five limitations across this set of PREA-related literature. First , there are many user-driven, policymaker/funder-commissioned reports and primarily descriptive approaches. A total of 28% of our literature set was explicitly policy/practice-oriented (i.e. policy report format) and 48% provided primarily thick descriptions of specific PREAs. Such literature is useful. However, user-oriented, thick descriptions alone seem insufficient to allow more critical perspectives and predictions regarding say, effects of arrangements and/or reactions (strategies, behaviours) of different organizational actors subjected to them (e.g. research funding agencies, universities, localized and more global knowledge communities). Similarly, descriptive accounts, even when oriented towards policy learning, may in fact hinder it because of a lack of analytical comparative foundations (and make it difficult to achieve ‘mutual learning’ across PREAs, as recommended by Sivertsen 2017 ). Descriptions of PREAs may make them appear comparable, transferable, or generalizable. Such comparisons are, however, often superficial. Lacking critical understanding of the use of whole or partial arrangements could lead to wide-ranging unintended and unexpected effects.

A second limitation is the pervasive, methodologically intractable unknowns in the literature concerning whether PREAs do produce, promote, or hinder research with specific performance-related properties (e.g. excellence, novelty, breakthrough, long-term focus, societal relevance, or impact). This is linked to a third limitation; the literature is inconclusive in answering whether—particularly after seeming early gains in using certain PREAs in specific countries—there are now increasing or diminishing returns for policymakers and universities to develop and deploy seemingly ever more expensive, extensive, and potentially intrusive arrangements.

A fourth limitation is that research on effects of PREAs has primarily focused on (self-)reported changes in universities. Reported effects—let alone causally attributable changes —to structures and organizations of national, trans-national, and trans-organizational research fields (knowledge communities, knowledge properties) have received much less attention. Most research has focused upon micro-level changes to research topics or topic portfolios pursued by researchers in specific universities, fields, and/or national systems.

It is clear that design and deployment of PREAs does not take place in a vacuum. PREAs are parts of and are strongly ‘coupled’ to a wider universe of path-dependent, dynamic activities, and exercise of power, authority, resources, politics, and policy machinery ( Whitley 2016 ). And yet a fifth limitation here is the absence of comparative frameworks to account for these aspects across the many and various development and use contexts of PREAs.

4.2 A novel research agenda on PREAs?

We believe four elements for a novel research agenda on PREAs emerge from our critical review and analysis of the state-of-the-art. First, very few, if any, analytical frameworks exist to study and compare research evaluation arrangements. There are examples of comparative frameworks ( Geuna and Martin 2003 ) but most, with a possible exception in Whitley, Glaser and Laudel (2018) use descriptive not analytical characteristics. This reduces analytical capacity and availability of heuristics and theory to explain the many interacting mechanisms present in and across micro, meso, and macro levels of the global science system. A novel research agenda could therefore first include development and testing of comparative analytical frameworks .

Literature on rationales for PREAs has predominantly dealt with efficiency concerns. They determine—we argue—whether arrangements have achieved what they set out to achieve. Studying efficiency of research evaluation as a policy instrument is a worthy pursuit. However, there are practical and analytical limitations inherent in this delineation of a research agenda. A second element of a novel research agenda would be to incorporate effectiveness concerns , that is, are the ‘right’ things being done in the science system? This should trace beyond localized conditions for research (e.g. at universities) to incorporate treatment of potential changes in the structure of global ‘research fields’ (c.f. Nedeva 2013 ).

Literature studying effects of PREAs on the science system has also largely focused on ad hoc associations between effects and measures. The arrangement under discussion is commonly taken to be a universal or singular enabler of the observed effects. A third element of a novel research agenda could be to attempt to add causal attribution to verify such assumptions.

Finally, we see from the literature that PREAs typically target research organizations in national policy and funding spaces. Correspondingly studies seemingly rarely study effects beyond those for universities in their own local context. 13 A fourth and final element of a novel research agenda would seem to be to include effects on the structure of global knowledge communities and bodies of knowledge . A summary of these five limitations and four novel agenda elements is provided in Table 3 .

Summary of limitations evident in and novel research agenda elements suggested by our critical review of PREA-related literature

Our critical review and purposive analysis of 354 pieces of literature we feel addresses the state-of-the-art on PREAs. It spanned works published from 1968 to 2018 and encompassed both scholarly and policy/practice-related research orientations. We believe our analysis satisfied our research aims, that is, to enable us to highlight key arguments, analyse limitations, and to suggest how to progress the research agenda in this area.

From our review we can conclude, first, analytical comparative frameworks are needed to study PREAs. Second, not only efficiency but also effectiveness concerns should be considered for PREAs. Third, studies should be devised and conducted on science system-level effects of PREAs and how global research fields are affected rather than just particular studies of local settings. Fourth, methodologies need to be advanced to measure and attribute these effects of PREAs on the (global) science system.

All four elements of this novel research agenda seem both necessary and challenging. There are numerous levels of mediation of effects and inherent complexities to unpack layer upon layer of research-related conditions here. We limited our article’s aims to (re-)opening the research agenda on PREAs and their effects on the science system by means of a critical, purposive, inductive examination of PREA-related research themes, and identification of agenda gaps. Developing analytical frameworks for PREAs, perhaps even outlining ‘ideal’ types of PREAs, and stretching studies of effects to include research fields appear essential. Similarly, learning to cope better with effectiveness, measurement and attribution issues seem necessary next steps, to take studies of PREAs further, to the benefit of both academic and practitioner interests.

We provide our full definition of ‘performance-based research evaluation arrangements’ in the following section, and distinguish it from research evaluation ‘systems’.

We focused our attention on journals publishing on topics of higher education studies, higher education policy, higher education management, sociology of science and science and technology policy studies, as well as fields like health policy and studies where research evaluation is addressed as a side issue in larger discussions (e.g. on priority setting). Indicative journals in our literature set include Cambridge Journal of Economics , Education Policy , Evidence & Policy , Higher Education , Higher Education Quarterly , Journal of Electronic Resources in Medical Libraries , Journal of Higher Education Policy and Management , Journal of Infometrics , Journal of Information Science , Journal of Sociology of Education , Journal of the Association for Information Science and Technology , Management and Policy , Minerva , Policy and Society , Political Studies Review , Public Administration , Public Management Review , Research Evaluation , Research Policy , Science, Science and Public Policy , Scientometrics , and Tertiary Education and Management .

All literature was in English except for one piece in Spanish. We did not try to access private, commercially sensitive, or confidential evaluations of specific research performers or funders. The entire set of academic and grey literature is heterogeneous, even though we confined our search to publicly available, English-language materials. This is likely due to significant involvement of funders in sponsoring research say, to audit their resource allocation processes and evaluate the outcomes of their funded research.

Our full country coverage includes Australia, Austria, Belgium/Flanders, Brazil, Bulgaria, Canada, China, Czech Republic, Denmark, Estonia, Finland, Germany, Hong Kong, Hungary, India, Ireland, Italy, Japan, Korea, Latvia, Lithuania, Mexico, Morocco, Netherlands, New Zealand, Norway, Poland, Romania, Slovak Republic, Slovenia, South Africa, Spain, Sweden, Switzerland, UK, USA, and Uruguay.

Correspondingly we cite some literature in multiple theme sections of our later findings, when their secondary message(s) are relevant, denoted by ‘see also’ in our citations. Our choice to allocate by primary theme rather than cluster in multiple themes by coverage of all issues is of course contentious. However, we believe this subjective approach provides a more useful thematic clustering for our purposes than exhaustively cataloguing by primary, secondary, tertiary, etc. themes.

We considered coding our subjective judgement of the apparent quality of literature. We decided against this step, in case it influenced our later analysis.

For ease of reference, we also included in our database columns for author name(s), title of the work, and publication year.

We provide numbers indicatively to show how much literature clustered into each theme, and the surveyed balance of approaches and content (e.g. scholarly, policy/practice-orientations). Our numbers and percentages do not constitute general impressions about the broader universe of evaluation-related research that exists outside our specific analytical boundaries for literature on PREAs.

One piece of literature we coded ‘other’; it was more abstract in its descriptive approach.

We considered ‘rationales’ for research evaluation to be within our analytical remit because they were present in the PREA-related literature. Our inductive clustering of themes reflects that these issues were being discussed in material within the scope of our PREA definition.

Of all the clustering themes, literature in Theme 3 was the most what we would call ‘synthetic’, in that primary messages often combined aspects of one or more of our analytical themes.

This ‘strategic approach’ concerns selectivity and concentration of research resources to research areas, researchers and teams, and universities displaying characteristics associated with excellence: share of highly cited publications, citations/impact, external grants capture, industry links, and patents. New tools and data to measure this notion of excellence have been associated with pressures on research systems to adapt to dominant ideas around value for money, steering and control, accountability, and measurement ( Butler 2003 ; see also Debackere and Glanzel 2003 ; Geuna and Martin 2003 ; Linkova 2014 ).

Other effects considered do include researcher careers, but predominantly just ‘organizational’ careers, still constraining analysis within the policy/funding/university ‘space’ and not on to ‘cognitive’ or ‘knowledge community’ careers in research fields (c.f. terminology from Laudel 2017 ).

We are grateful to two anonymous referees for their insightful feedback.

This work was supported by the Swedish Foundation for Social Science and Humanities Research (Riksbankens Jubileumsfond) [FSK15-0881:1]. Knowledge in science and policy: creating an evidence base for converging modes of governance in policy and science (KNOWSCIENCE).

Conflict of interest statement . None declared.

Aagaard K. ( 2015 ) ‘ How Incentives Trickle Down: Local Use of a National Bibliometric Indicator System ’, Science and Public Policy , 42 : 725 – 37 .

Google Scholar

Aagaard K. , Bloch C. , Schneider J. W. ( 2015 ) ‘ Impacts of Performance-Based Research Funding Systems: The Case of the Norwegian Publication Indicator ’, Research Evaluation , 24 : 106 – 17 .

Abramo G. , Cicero T. , D’Angelo A. C. ( 2013 ) ‘ National Peer-Review Research Assessment Exercises for the Hard Sciences Can Be a Complete Waste of Money: The Italian Case ’, Scientometrics , 95 : 311 – 24 .

Abramo G. , D’Angelo C. A. ( 2011 ) ‘ Evaluating Research: From Informed Peer Review to Bibliometrics ’, Scientometrics , 87 : 499 – 514 .

Abramo G. , D’Angelo C. A. , Di Costa F. ( 2008 ) ‘ Assessment of Sectoral Aggregation Distortion in Research Productivity Measurements ’, Research Evaluation , 17 : 111 – 21 .

Abramo G. , D’Angelo C. A. , Di Costa F. ( 2011 ) ‘ National Research Assessment Exercises: The Effects of Changing the Rules of the Game during the Game ’, Scientometrics , 88 : 229 – 38 .

Adams J. , Gurney K. ( 2010 ) Funding Selectivity, Concentration and Excellence—How Good Is the UK’s Research? < http://www.rin.ac.uk/system/files/attachments/Funding_selectivity_concentration__excellence_-_Exec_Summ.pdf > accessed 28 Mar 2019.

Adler P. A. , Adler P. ( 1987 ) Membership Roles in Field Research (SAGE University Paper Series on Qualitative Research, Vol. 6) . Newbury Park, CA : SAGE .

Google Preview

Agyemang G. , Broadbent J. ( 2015 ) ‘ Management Control Systems and Research Management in Universities. An Empirical and Conceptual Exploration ’, Accounting, Auditing & Accountability Journal , 28 : 1018 – 46 .

Ancaiani A. , Anfossi A. F. , Barbara A. , Benedetto S. , Blasi B. , Carletti V. , Cicero T. , Ciolfi A. , Costa F. , Colizza G. , Costantini M. , di Cristina F. , Ferrara A. , Lacatena R. M. , Malgarini M. , Mazzotta I. , Nappi C. A. , Romagnosi S. , Sileoni S. ( 2015 ) ‘ Evaluating Scientific Research in Italy: The 2004–10 Research Evaluation Exercise ’, Research Evaluation , 24 : 242 – 55 .

Arnold E. , Mahieu B. ( 2015 ) R&D Evaluation Methodology and Funding Principles. Summary Report . Technopolis. < https://www.slideshare.net/ipnmetodika/2104-summary-report> accessed 28 Mar 2019.

Arnold E., et al. . ( 2014 ) The Role of Metrics in Performance-Based Research Funding Systems. A Report to the Russell Group . Brighton, UK : Technopolis Group .

Arocena R. , Göransson B. , Sutz J. ( 2018 ) ‘ Towards Making Research Evaluation More Compatible with Developmental Goals ’, Science and Public Policy , 46 : 210 – 8 .

Auranen O. , Nieminen M. ( 2010 ) ‘ University Research Funding and Publication Performance—An International Comparison ’, Research Policy , 39 : 822 – 34 .

Barker K. ( 2007 ) ‘ The UK Research Assessment Exercise: The Evolution of a National Research Evaluation System ’, Research Evaluation , 16 : 3 – 12 .

Bence V. , Oppenheim C. ( 2005 ) ‘ The Evolution of the UK’s Research Assessment Exercise: Publications, Performance and Perceptions ’, Journal of Educational Administration and History , 37 : 137 – 55 .

Benner M. , Sandström U. ( 2000 ) ‘ Institutionalizing the Triple Helix: Research Funding and Norms in the Academic System ’, Research Policy , 29 : 291 – 301 .

Bertocchi G. , Gambardella A. , Jappelli T. , Nappi C. A. , Peracchi F. ( 2015 ) ‘ Bibliometric Evaluation vs. Informed Peer Review: Evidence from Italy ’, Research Policy , 44 : 451 – 66 .

Bianco M. , Gras N. , Sutz J. ( 2016 ) ‘ Academic Evaluation: Universal Instrument? Tool for Development? ’, Minerva , 54 : 399 – 421 .

Bozeman B. , Dietz J. , Gaughan M. ( 2001 ) ‘ Scientific and Technical Human Capital: An Alternative Model for Research Evaluation ’, International Journal of Technology Management , 7 : 716 – 40 .

Bozeman B. , Sarewitz D. ( 2011 ) ‘ Public Value Mapping and Science Policy Evaluation ’, Minerva , 49 : 1 – 23 .

Broadbent J. ( 2010 ) ‘ The UK Research Assessment Exercise: Performance Measurement and Resource Allocation ’, Australian Accounting Review , 52 : 14 – 23 .

Butler L. ( 2003 ) ‘ Explaining Australia’s Increased Share of ISI Publications—The Effects of a Funding Formula Based on Publication Counts ’, Research Policy , 32 : 143 – 55 .

Butler L. ( 2007 ) ‘ Assessing University Research: A Plea for a Balanced Approach ’, Science and Public Policy , 34 : 565 – 74 .

Butler L. ( 2008 ) ‘ Using a Balanced Approach to Bibliometrics: Quantitative Performance Measures in the Australian Research Quality Framework ’, Ethics in Sciences and Environmental Politics , 8 : 83 – 92 .

Campbell M. , Boxall M. ( 2004 ) Better Accountability Revisited: Review of Accountability Costs 2004 . PA Consulting Group. < https://dera.ioe.ac.uk/4985/1/rd06_04.pdf> accessed 28 Mar 2019.

Canibano C. , Vilardell I. , Corona C. , Benito-Amat C. ( 2018 ) ‘ The Evaluation of Research Excellence and the Dynamics of Knowledge Production in the Humanities: The Case of History in Spain ’, Science and Public Policy , 45 : 775 – 89 .

Cattaneo M. , Meoli M. , Signori A. ( 2016 ) ‘ Performance-Based Funding and University Research Productivity: The Moderating Effect of University Legitimacy ’, The Journal of Technology Transfer , 41 : 85 – 104 .

Claeys-Kulik A. L. , Estermann T. ( 2015 ) Define Thematic Report: Performance-Based Funding of Universities in Europe . European University Association. < https://eua.eu/downloads/publications/define%20thematic%20report%20performance-based%20funding%20of%20universities%20in%20europe.pdf > accessed 28 Mar 2019.

Court S. ( 1999 ) ‘ Negotiating the Research Imperative: The Views of UK Academics on Their Career Opportunities ’, Higher Education Quarterly , 53 : 65 – 87 .

Cozzens S. E. ( 1981 ) ‘ Taking the Measure of Science: A Review of Citation Theories ’, Newsletter of the International Society for the Sociology of Knowledge , 7 : 16 – 21 .

Cruz-Castro L. , Sanz-Menéndez L. ( 2007 ). ‘Research Evaluation in Transition: Individual versus Organisational Assessment in Spain’, in Whitley Richard and Gläser Jochen (eds) The Changing Governance of the Sciences: The Advent of Research Evaluation Systems , pp. 205 – 23 . Dordrecht : Kluwer .

Cunningham P. , Salavetz A. , Tuytens P. ( 2012 ) Monitoring Social Sciences and Humanities . Metris. Synthesis Report 2012. Brighton, UK: Technopolis Group.

De Boer H. , Jongbloed B. W. A. , Benneworth P. S. , Cremonini L. , Kolster R. , Kottmann A. , Lemmens-Krug K. , Vossensteyn J. J. ( 2015 ) Performance-Based Funding and Performance Agreements in Fourteen Higher Education Systems. Report for the Ministry of Education, Culture and Science . Netherlands: Centre for Higher Education Policy Studies, Universiteit Twente.

De Jong S. , van Arensbergen P. , Daemen F. F. , van der Meulen B. , van den Besselaar P. ( 2011 ) ‘ Evaluation of Research in Context: An Approach and Two Cases ’, Research Evaluation , 20 : 61 – 72 .

De Rijcke S. , Wouters P. F. , Rushforth A. D. , Franssen T. P. , Hammarfelt B. ( 2016 ) ‘ Evaluation Practices and Effects of Indicator Use—A Literature Review ’, Research Evaluation , 25 : 161 – 9 .

Debackere K. , Glanzel W. ( 2003 ) Using a Bibliometric Approach to Support Research Policy Decisions: The Case of the Flemish BOF-Key. Research Report 0306, D/2003/2376/ 06 : 1 – 26 . Leuven, Belgium: Katholieke Universiteit Leuven.

Demiris G. , Oliver D. P. , Washington K. T. ( 2019 ) ‘Defining and Analyzing the Problem’, in Demiris G. , Oliver D. P. , Washington K. T. (eds) Behavioral Intervention Research in Hospice and Palliative Care , pp. 27 – 39 . Amsterdam, Netherlands : Elsevier Science and Technology Academic Press .

Donovan C. ( 2007 ) ‘ The Qualitative Future of Research Evaluation ’, Science and Public Policy , 34 : 585 – 97 .

Donovan C. ( 2008 ) ‘The Australian Research Quality Framework: A Live Experiment in Capturing the Social, Economic, Environmental, and Cultural Returns of Publicly Funded Research’, in Coryn C. , Scriven M. (eds) Reforming the Evaluation of Research. New Directions for Evaluation , pp. 47 – 60 . San Francisco, CA : Jossey-Bass .

Donovan C. ( 2009 ) ‘ Gradgrinding the Social Sciences: The Politics of Metrics of Political Science ’, Political Studies Review , 7 : 73 – 83 .

Elsevier ( 2011 ) International Comparative Performance of the UK Research Base. A Report Prepared for the Department of Business, Innovation and Skills . < https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/32489/11-p123-international-comparative-performance-uk-research-base-2011.pdf > accessed 28 Mar 2019.

Elsevier ( 2013 ) International Comparative Performance of the UK Research Base. A Report Prepared for the Department of Business, Innovation and Skills . < https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/263729/bis-13-1297-international-comparative-performance-of-the-UK-research-base-2013.pdf > accessed 28 Mar 2019.

Elton L. ( 2000 ) ‘ The UK Research Assessment Exercise: Unintended Consequences ’, Higher Education Quarterly , 54 : 274 – 83 .

European Centre for Strategic Management of Universities (ESMU) ( 2010 ) Funding Higher Education: A View across Europe . < https://uniko.ac.at/modules/download.php? key=4488_DE_O&cs=8E8 > accessed 28 Mar 2019.

European Commission ( 2009 ) The Role of Community Research Policy in the Knowledge-Based Economy. Experts Group Report . Belgium. < https://publications.europa.eu/en/publication-detail/-/publication/8edc4431-c8e7-49ba-a98a-ba0dc911ca13/language-en > accessed 28 Mar 2019.

Eyre-Walker A. , Stoletzki N. ( 2013 ) ‘ The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citation ’, PLoS Biology , 11 : e1001675 .

Ferlie E. , Musselin C. , Andresani G. ( 2008 ) ‘ The Steering of Higher Education Systems: A Public Management Perspective ’, Higher Education , 56 : 325 – 48 .

Fiala D. ( 2013 ) ‘ Science Evaluation in the Czech Republic: The Case of Universities ’, Societies , 3 : 1 – 13 .

Franceschet M. , Costantini A. ( 2011 ) ‘ The First Italian Research Assessment Exercise: A Bibliometric Perspective ’, Journal of Infometrics , 5 : 275 – 91 .

Frankel M. , Goddard A. , Ransow G. ( 2014 ) ‘ Golden Triangle Pulls Ahead in REF Shake-out. UCL and KCL Ascend Power Rankings, Manchester and Leeds Fall ’, Research Fortnight, Issue , 4470 : 1 – 20 .

Franzoni C. , Scellato G. , Stephan P. ( 2011 ) ‘ Science Policy: Changing Incentives to Publish ’, Science , 333 : 702 – 3 .

Frølich N. ( 2008 ) The Politics of Steering by Numbers: Debating Performance-Based Funding in Europe . Oslo, Norway : NIFU STEP .

Frølich N. ( 2011 ) ‘ Multi-Layered Accountability. Performance-Based Funding of Universities ’, Public Administration , 89 : 840 – 59 .

Frølich N. , Schmidt E. , Rosa M. J. ( 2010 ) ‘ Funding Systems for Higher Education and Their Impacts on Institutional Strategies and Academia. A Comparative Perspective ’, International Journal of Educational Management , 24 : 7 – 21 .

Galleron I. , Ochsner M. , Spaapen J. , Williams G. ( 2017 ) ‘ Valorizing SSH Research: Towards a New Approach to Evaluate SSH Research’ Value for Society ’, Fteval Journal for Research and Technology Policy Evaluation , 44 : 35 – 41 .

Geuna A. , Martin B. R. ( 2003 ) ‘ University Research Evaluation and Funding: An International Comparison ’, Minerva , 41 : 277 – 304 .

Geuna A. , Piolatto M. ( 2016 ) ‘ The Development of Research Assessment in the UK and Italy: Costly and Difficult, but Probably Worth (for a While) ’, Research Policy , 45 : 260 – 71 .

Gibson J. , Anderson D. L. , Tressler J. ( 2014 ) ‘ Which Journal Rankings Best Explain Academic Salaries? Evidence from the University of California ’, Economic Inquiry , 52 : 1322 – 40 .

Gläser J. ( 2007 ) ‘The Social Orders of Research Evaluation Systems’, in Whitley Richard and Gläser Jochen (eds) The Changing Governance of the Sciences: The Advent of Research Evaluation Systems , pp. 245 – 66 . Dordrecht : Springer .

Gläser J. ( 2019 ) ‘How Can Governance Change Research Content? Linking Social Policy Studies to the Sociology of Science’, in Simon D. , Kuhlmann S. , Stamm J. , Canzler W. (eds) Handbook on Science and Public Policy , pp. 419 – 47 . Cheltenham, UK and Northampton, MA, USA : Edward Elgard .

Gläser J. , Lange S. , Laudel G. , Schimank U. ( 2010 ) ‘Informed Authority? The Limited Use of Research Evaluation Systems for Managerial Control in Universities’, in Whitley R. , Gläser J. , Engwall L. (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and Their Consequences for Intellectual Innovation , pp. 149 – 369 . Oxford : Oxford University Press .

Grant J. ( 2010 ) Capturing Research Impacts. A Review of International Practice . RAND Europe. < https://www.rand.org/content/dam/rand/pubs/documented_briefings/2010/RAND_DB578.pdf> accessed 28 Mar 2019.

Greenhalgh T. , Fahy N. ( 2015 ) ‘ Research Impact in the Community-Based Health Sciences: An Analysis of 162 Case Studies from the 2014 UK Research Excellence Framework ’, BMC Medicine , 13 : 1 – 12 .

Hamann J. ( 2016 ) ‘ The Visible Hand of Research Performance Assessment ’, Higher Education , 72 : 761 – 79 .

Hammarfelt B. , de Rijcke S. ( 2015 ) ‘ Accountability in Context: Effects of Research Evaluation Systems on Publication Practices, Disciplinary Norms, and Individual Working Routines in the Faculty of Arts at Uppsala University ’, Research Evaluation , 24 : 63 – 77 .

Hare P. G. ( 2003 ) ‘ The United Kingdom’s Research Assessment Exercise: Impact on Institutions, Departments, Individuals ’, Higher Education Management and Policy , 15 : 43 – 62 .

Harland T. , Tidswell T. , Everett D. , Hale L. , Pickering N. ( 2010 ) ‘ Neoliberalism and the Academic as Critic and Conscience of Society ’, Teaching in Higher Education , 15 : 185 – 96 .

Higher Education Funding Council for England (HEFCE) ( 2014 ) A Review of QR Funding in English HEIs. Process and Impact . < https://www.praxisauril.org.uk/sites/praxisunico.org.uk/files/HEFCE_2014_qrreview.pdf > accessed 28 Mar 2019.

Henkel M. ( 1999 ) ‘ The Modernisation of Research Evaluation: The Case of the UK ’, Higher Education , 38 : 105 – 22 .

Hicks D. ( 2010 ) ‘Overview of Models of Performance-Based Research Funding Systems’, in OECD, Performance Based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings , 1 December, pp. 23–52. Paris, France: OECD Publishing.

Hicks D. ( 2012 ) ‘ Performance-Based University Research Funding Systems ’, Research Policy , 41 : 251 – 61 .

Himanen L. , Auranen O. , Puuska H.-M. , Nieminen M. ( 2009 ) ‘ Influence of Research Funding and Science Policy on University Research Performance: A Comparison of Five Countries ’, Science and Public Policy , 36 : 419 – 30 .

Holmes R. ( 2015 ) ‘ My Tongue on Your Theory: The Bittersweet Reminder of Every-Thing Unnameable ’, Discourse, Studies in the Cultural Politics of Education , 37 : 662 – 79 .

Hug S. E. , Ochsner M. , Daniel H.-D. ( 2013 ) ‘ Criteria for Assessing Research Quality in the Humanities: A Delphi Study among Scholars of English Literature, German Literature and Art History ’, Research Evaluation , 22 : 369 – 83 .

Hughes A. , Kitson M. , Bullock A. ( 2013 ) The Dual Funding Structure for Research in the UK: Research Council and Funding Council Allocation Methods and the Pathways to Impact of UK Academics . Research report issued 5 April. Cambridge and London, UK: University of Cambridge and Imperial College London.

Ioannidis J. ( 2018 ) ‘ Meta-Research: Why Research on Research Matters ’, PLoS Biology , 16 : e2005468 .

Iorwerth A. ( 2005 ) ‘Methods of Evaluating University Research around the World’. Working Paper 2005-04. Department of Finance, Canada.

Johnston R. ( 1994 ) ‘ Effects of Resource Concentration on Research Performance ’, Higher Education , 28 : 25 – 37 .

Jonkers K. , Sachwald F. ( 2018 ) ‘ The Dual Impact of ‘Excellent’ Research on Science and Innovation: The Case of Europe ’, Science and Public Policy , 45 : 159 – 74 .

Jonkers K. , Zacharewicz T. ( 2015 ) Performance Based Funding: A Comparative Assessment of Their Use and Nature in EU Member States . EUR 27477. Brussels, Belgium: European Union JRC Science Hub, JRC97684.

Kallerud E. , Finnbjørnsson T. , Geschwind L. , Häyrinen-Alestalo M. , Ramberg I. , Siune K. , Tuominen T. ( 2011 ) Public Debate on Research Policy in the Nordic Countries. A Comparative Analysis of Actors and Issues (1998–2007) . < http://www.nifu.no/Norway/Publications/2011/Webrapport%2011-2011.pdf > accessed 28 Mar 2019.

Kaufmann A. , Kasztler A. ( 2009 ) ‘ Differences in Publication and Dissemination Practices between Disciplinary and Transdisciplinary Science and the Consequences for Research Evaluation ’, Science and Public Policy , 36 : 215 – 27 .

Kenna R. , Berche B. ( 2011 ) ‘ Normalization of Peer-Evaluation Measures of Group Research Quality across Academic Disciplines ’, Research Evaluation , 20 : 107 – 16 .

Knowles C. , Burrows R. ( 2014 ) ‘ The Impact of Impact ’, Etnográfica , 18 : 237 – 54 .

Kwok J. T. ( 2013 ) Impact of ERA Research Assessment on University Behaviour and Their Staff . National Tertiary Education Union. < https://core.ac.uk/download/pdf/30677048.pdf > accessed 28 Mar 2019.

Langfeldt L. , Nedeva M. , Sörlin S. , Thomas D. A. ( 2020 ) ‘ Co-Existing Notions of Research Quality: A Framework to Study Context-Specific Understanding of Good Research ’, Minerva , 58 : 115 – 137 .

Laudel G. ( 2005 ) ‘ Quality-Only Assumption. Is External Research Funding a Valid Indicator for Research Performance? ’, Research Evaluation , 14 : 27 – 34 .

Laudel G. ( 2017 ) ‘ How Do National Career Systems Promote or Hinder the Emergence of New Research Lines? ’, Minerva , 55 : 341 – 69 .

Lee F. S. , Harley S. ( 1998 ) ‘ Peer Review, the Research Assessment Exercise and the Demise of Non-Mainstream Economics ’, Capital & Class , 66 : 23 – 51 .

Lee F. S. , Pham X. , Gu G. ( 2013 ) ‘ The UK Research Assessment Exercise and the Narrowing of UK Economics ’, Cambridge Journal of Economics , 37 : 693 – 717 .

Leisyte L. , Westerheijden D. F. ( 2014 ) Research Evaluation and Its Implications for Academic Research in the United Kingdom and the Netherlands . Discussion paper No. 1-2014. Dortmund, Germany: Technische Universität Dortmund.

Lepori B. , Masso J. , Jabłecka J. , Sima K. , Ukrainski K. ( 2009 ) ‘ Comparing the Organization of Public Research Funding in Central and Eastern European Countries ’, Science and Public Policy , 36 : 667 – 81 .

League of European Research Universities (LERU) ( 2012 ) Research Universities and Research Assessment . < https://www.leru.org/files/Research-Universities-and-Research-Assessment-Full-paper.pdf > accessed 28 Mar 2019.

Lind J. ( 2019 ) ‘ The Missing Link: How University Managers Mediate the Impact of a Performance-Based Research Funding System ’, Research Evaluation , 28 : 84 – 93 .

Linkova M. ( 2014 ) ‘ Unable to Resist: Researchers’ Responses to Research Assessment in the Czech Republic ’, Human Affairs , 24 : 78 – 88 .

Luo J. , Ordóñez-Matamoros G. , Kuhlmann S. ( 2019 ) ‘ The Balancing Role of Evaluation Mechanisms in Organizational Governance—The Case of Publicly Funded Research Institutions ’, Research Evaluation , 28 : 344 – 54 .

Luukkonen T. ( 1997 ) ‘ Why Has Latour’s Theory of Citations Been Ignored by the Bibliometric Community? Discussion of Sociological Interpretations of Citation Analysis ’, Scientometrics , 38 : 27 – 37 .

Luukkonen T. , Thomas D. A. ( 2016 ) ‘ The ‘Negotiated Space’ of University Researchers’ Pursuit of a Research Agenda ’, Minerva , 54 : 99 – 127 .

Macdonald S. , Kam J. ( 2010 ) ‘ Counting Footnotes: Citability in Management Studies ’, Scandinavian Journal of Management , 26 : 189 – 203 .

Mahieu B. , Arnold E. , Kolarz P. ( 2013 ) Measuring Scientific Performance for Improved Policy Making. Literature Review . Brighton, UK : Technopolis Group .

Mahieu B. , Arnold E. , Kolarz P. ( 2014 ) Measuring Scientific Performance for Improved Policy Making. Final Report—Summary (D6) . Brighton, UK : Technopolis Group .

Mahieu B. , Arnold E. ( 2015 ) R&D Evaluation Methodology and Funding Principles. Final Report 1: The R&D Evaluation Methodology . Brighton, UK : Technopolis Group .

Mahieu B., Brown, N., Fridholm, T., Giarracca, F., Hinojosa, C., Horvath, A., Potau, X., Quas, A., and Tummers, A et al.  ( 2013 ) Measuring Scientific Performance for Improved Policy-Making. Current Practice in the EU Member States (D3—Survey Report) . Brighton, UK : Technopolis Group .

Martin B. , Whitley R. ( 2010 ) ‘The UK Research Assessment Exercise: A Case of Regulatory Capture?’, in Whitley R. , Glaser J. , Engwall L. (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and Their Consequences for Intellectual Innovation , pp. 51 – 80 . Oxford, UK: Oxford University Press .

Martin B. R. ( 2011 ) ‘ The Research Excellence Framework and the ‘Impact Agenda’: Are we Creating a Frankenstein Monster? ’, Research Evaluation , 20 : 247 – 54 .

Martin B. R. ( 2016 ) ‘What is Happening to Our Universities?’ SPRU Working Paper Series 2016-03, 1-26. Brighton, UK: SPRU. ISSN 2057 – 6668 .

Merton R. K. ( 1968 ) ‘ The Matthew Effect in Science. The Reward and Communication Systems of Science Are Considered ’, Science , 159 : 56 – 63 .

Mingers J. , White L. ( 2015 ) Throwing Out the Baby with the Bathwater: The Undesirable Effects of National Research Assessment Exercises on Research . < https://arxiv.org/ftp/arxiv/papers/1502/1502.00658.pdf > accessed 28 Mar 2019.

Ministry of Education ( 2012 ) An International Comparison of Performance-Based Research Funding Systems (PBRFS) . Wellington, New Zealand: New Zealand Parliament Library.

Mok K. H. ( 2014 ) ‘ Enhancing Quality of Higher Education for World-Class Status ’, Chinese Education & Society , 47 : 44 – 64 .

Morris N. ( 2010 ) ‘Authority Relations as Conditions for, and Outcome of, Shifts of Governance: The Limited Impact of the UK Research Assessment Exercise on the Biosciences’, in Whitley R. , Glaser J. , Engwall L. (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and Their Consequences for Intellectual Innovation , pp. 239 – 64 . Oxford, UK: Oxford University Press .

Murphy T. , Sage D. ( 2014 ) ‘ Perceptions of the UK’s Research Excellence Framework 2014: A Media Analysis ’, Journal of Higher Education Policy and Management , 36 : 603 – 15 .

Nedeva M. ( 2013 ) ‘ Between the Global and the National: Organising European Science ’, Research Policy , 42 : 220 – 30 .

Nedeva M. , Braun D. , Edler J. , Glaser J. , Laredo P. , Laudel G. , Luukkonen T. , Stampfer M. , Thomas D. , Whitley R. ( 2012 ) Understanding and Assessing the Impact and Outcomes of the ERC and Its Funding Schemes, EURECIA Final Synthesis Report . < http://erc.europa.eu/sites/default/files/document/file/eurecia_final_synthesis_report.pdf > accessed 28 Mar 2019.

Neufeld J. , von Ins M. ( 2011 ) ‘ Informed Peer Review and Uninformed Bibliometrics? ’, Research Evaluation , 20 : 31 – 46 .

Ochsner M. , Hug S. E. , Daniel H.-D. ( 2012 ) ‘ Indicators for Research Quality for Evaluation of Humanities Research: Opportunities and Limitations ’, Bibliometrie - Praxis und Forschung , 4 : 1 – 17 .

Ohtani R. , Kamo M. , Kobayashi N. ( 2013 ) ‘ An Introduction to the Research Excellence Framework: A New Research Evaluation Framework for Universities in the UK—A Comparison with the Status of Research Evaluation in Japan ’, Synthesiology , 6 : 118 – 25 .

Olson S. , Rapporteurs S. M. ( 2011 ) Measuring the Impacts of Federal Investments in Research. A Workshop Summary . USA : National Academy of Sciences .

Olssen M. ( 2016 ) ‘ Neoliberal Competition in Higher Education Today: Research, Accountability and Impact ’, British Journal of Sociology of Education , 37 : 129 – 48 .

Organisation for Economic Co-operation and Development (OECD) ( 2009 ) Enhancing Public Research Performance through Evaluation, Impact Assessment and Priority Setting . < https://www.oecd.org/sti/inno/Enhancing-Public-Research-Performance.pdf> accessed 28 Mar 2019.

Organisation for Economic Co-operation and Development (OECD) ( 2010 a) Performance-Based Funding for Public Research in Tertiary Education Institutions . Australia: Web annex: additional country detail.

Organisation for Economic Co-operation and Development (OECD) ( 2010 b) Performance-Based Funding for Public Research in Tertiary Education Institutions Workshop Proceedings . < https://www.oecd-ilibrary.org/fr/education/performance-based-funding-for-public-research-in-tertiary-education-institutions_9789264094611-en > accessed 28 Mar 2019.

Organisation for Economic Co-operation and Development (OECD) ( 2011 ) Issue Brief: Public Sector Research Funding . < http://www.oecd.org/innovation/policyplatform/48136600.pdf > accessed 28 Mar 2019.

Orr D. ( 2004 ) ‘ Research Assessment as an Instrument for Steering Higher Education. A Comparative Study ’, Journal of Higher Education Policy and Management , 26 : 345 – 62 .

Otley D. ( 2010 ) ‘ Research Assessment in the UK: An Overview of 1992–2008 ’, Australian Accounting Review , 52 : 3 – 13 .

PA Consulting Group ( 2008 ) RAE 2008 Accountability Review . London, UK: PA Consulting.

Paradeise C. , Thoenig J.-C. ( 2015 ) Search of Academic Quality . Houndmills, Basingstoke : Palgrave Macmillan .

Reale E. , Avramov D. , Canhial K. , Donovan C. , Flecha R. , Holm P. , Larkin C. , Lepori B. , Mosoni-Fried J. , Oliver E. , Primeri E. , Puigvert L. , Scharnhorst A. , Schubert A. , Soler M. , Soòs S. , Sordé T. , Travis C. , Van Horik R. ( 2018 ) ‘ A Review of Literature on Evaluating the Scientific, Social and Political Impact of Social Sciences and Humanities Research ’, Research Evaluation , 27 : 298 – 308 .

Rebora G. , Turri M. ( 2013 ) ‘ The UK and Italian Research Assessment Exercises Face to Face ’, Research Policy , 42 : 1657 – 66 .

Regan A. , Henchion M. ( 2019 ) ‘ Making Sense of Altmetrics: The Perceived Threats and Opportunities for Academic Identity ’, Science and Public Policy , 46 : 479 – 89 .

Rip A. , van der Meulen B. ( 1996 ) ‘ The Post-Modern Research System ’, Science and Public Policy , 23 : 343 – 52 .

Roberts P. ( 2007 ) ‘ Neoliberalism, Performativity and Research ’, Review of Education , 53 : 349 – 65 .

Sandström U. , Van den Besselaar P. ( 2018 ) ‘ Funding, Evaluation, and the Performance of National Research Systems ’, Journal of Infometrics , 12 : 365 – 84 .

Sastry T. , Bekhradnia B. ( 2014 ) Using Metrics to Allocate Research Funds. A Short Evaluation of Alternatives to the Research Assessment Exercise . Higher Education Policy Institute. < https://www.hepi.ac.uk/wp-content/uploads/2014/02/23RAEandmetricsfullreport.pdf > accessed 28 Mar 2019.

Sivertsen G. ( 2017 ) ‘ Unique, but Still Best Practice? The Research Excellence Framework (REF) from an International Perspective ’, Palgrave Communications , 3 : 17078 .

Smith S. , Ward V. , House A. ( 2011 ) ‘ ‘Impact’ in the Proposals for the UK’s Research Excellence Framework: Shifting the Boundaries of Academic Autonomy ’, Research Policy , 40 : 1369 – 79 .

Sørensen M. P. , Bloch C. , Young M. ( 2015 ) ‘ Excellence in the Knowledge-Based Economy: From Scientific to Research Excellence ’, European Journal of Higher Education , 6 : 217 – 36 .

Sörlin S. ( 2007 ) ‘ Funding Diversity: Performance-Based Funding Regimes as Drivers of Differentiation in Higher Education Systems ’, Higher Education Policy , 20 : 413 – 40 .

Sousa S. B. , Brennan J. L. ( 2014 ) ‘The UK Research Excellence Framework and the Transformation of Research Production’, in Musselin C. , Teixeira P. (eds) Reforming Higher Education , pp. 65 – 80 . Dordrecht : Springer .

Spaapen J. , Van Drooge L. ( 2011 ) ‘ Introducing ‘Productive Interactions’ in Social Impact Assessment ’, Research Evaluation , 20 : 211 – 8 .

Strehl F. , Reisinger S. , Kalatschan M. ( 2007 ) ‘Funding Systems and their Effects on Higher Education Systems’, OECD Education Working Paper, No. 6. Paris, France: OECD Publishing.

Tahar S. , Boutellier R. ( 2013 ) ‘ Resource Allocation in Higher Education in the Context of New Public Management ’, Public Management Review , 15 : 687 – 711 .

Technopolis ( 2009 ) Identification and Dissemination of Lessons Learned by Institutions Participating in the Research Excellence Framework (REF) Bibliometrics Pilot. Results of the Round One Consultation . Brighton, UK: Technopolis.

Technopolis ( 2010 ) REF Research Impact Pilot Exercise Lessons-Learned Project: Feedback on Pilot Submissions. Final Report . Brighton, UK: Technopolis.

Van der Meulen B. ( 1998 ) ‘ Science Policies as Principal–Agent Games Institutionalization and Path Dependency in the Relation between Government and Science ’, Research Policy , 27 : 397 – 414 .

Van Drooge L. , de Jong S. , Faber M. , Westerheijden D. ( 2013 ) Twenty Years of Research Evaluation . The Rathenau Instituut. < https://www.rathenau.nl/sites/default/files/2018-05/Twenty_years_of_research_evaluation_-_Rathenau_01.pdf > accessed 28 Mar 2019.

Vincent A. ( 2015 ) ‘ The Ideological Context of Impact ’, Political Studies Review , 13 : 474 – 84 .

Waitere H. J. , Wright J. , Tremaine M. , Brown S. , Pause C. J. ( 2011 ) ‘ Choosing Whether to Resist or Reinforce the New Managerialism: The Impact of Performance‐Based Research Funding on Academic Identity ’, Higher Education Research & Development , 30 : 205 – 17 .

Wang J. , Hicks D. ( 2012 ) ‘Policy screening by structural change detection: can policies effectively boost research system performance?’, Proceedings of 17th International Conference on Science and Technology Indicators (2012). < http://works.bepress.com/diana_hicks/35/ > accessed 28 Mar 2019.

Watermeyer R. ( 2014 ) ‘ Issues in the Articulation of ‘Impact’: The Responses of UK Academics to ‘Impact’ as a New Measure of Research Assessment ’, Studies in Higher Education , 39 : 359 – 77 .

Watermeyer R. ( 2016 ) ‘ Impact in the REF: Issues and Obstacles ’, Studies in Higher Education , 41 : 199 – 214 .

Whitley R. ( 2011 ) ‘ Changing Governance and Authority Relations in the Public Sciences ’, Minerva , 49 : 359 – 85 .

Whitley R. ( 2016 ) ‘ Varieties of Scientific Knowledge and Their Contributions to Dealing with Policy Problems: A Response to Richard Nelson’s “the Sciences Are Different and the Differences Matter” ’, Research Policy , 45 : 1702 – 7 .

Whitley R. , Gläser J. (eds) ( 2007 ) The Changing Governance of the Sciences. The Advent of Research Evaluation Systems . The Netherlands : Springer .

Whitley R. , Glaser J. , Laudel G. ( 2018 ) ‘ The Impact of Changing Funding and Authority Relationships on Scientific Innovations ’, Minerva , 56 : 109 – 34 .

Wilczynski S. M. ( 2017 ) ‘Other Sources of Evidence’, in Wilczynski S. M. (ed) Critical Specialities—Treating Autism & Behavioral Challenge , pp. 13 – 19 . Amsterdam, Netherlands : Elsevier Science and Technology Academic Press .

Wilsdon J. R. , Allen L. , Belfiore E. , Campbell P. , Curry S. , Hill S. , Jones R. A. L. , Kain R. , Kerridge S. , Thelwall M. , Tinkler J. , Viney I. , Wouters P. , Hill J. , Johnson B. ( 2015 ) The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management . DOI: 10.13140/RG.2.1.4929.1363.

Woelert P. ( 2015 ) ‘ The ‘Logic of Escalation’ in Performance Measurement: An Analysis of the Dynamics of a Research Evaluation System ’, Policy and Society , 34 : 75 – 85 .

Woelert P. , McKenzie L. ( 2018 ) ‘ Follow the Money? How Australian Universities Replicate National Performance-Based Funding Mechanisms ’, Research Evaluation , 27 : 184 – 95 .

Wooding S. , Grant J. ( 2003 ) Assessing Research: The Researchers’ View . MR-1698-HEFCE. UK: HEFCE and RAND.

Wouters P. , Thelwall M. , Kousha K. , Waltman L. , de Rijcke S. , Rushforth A. , Franssen T. ( 2015 ) The Metric Tide: Literature Review. Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management . UK: HEFCE.

Wunsch-Vincent S. ( 2012 ) ‘Accounting for Science-Industry Collaboration in Innovation: Existing Metrics and Related Challenges’, in WIPO, Global Innovation Index 2012: Stronger Innovation Linkages for Global Growth , pp. 97–107. < https://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2012-chapter4.pdf > accessed 28 Mar 2019.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Organizing Your Social Sciences Research Assignments

• Annotated Bibliography
• Analyzing a Scholarly Journal Article
• Group Presentations
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• Types of Structured Group Activities
• Group Project Survival Skills
• Leading a Class Discussion
• Multiple Book Review Essay
• Reviewing Collected Works
• Writing a Case Analysis Paper
• Writing a Case Study
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Reflective Paper
• Writing a Research Proposal
• Generative AI and Writing
• Acknowledgments

Definition and Introduction

Journal article analysis assignments require you to summarize and critically assess the quality of an empirical research study published in a scholarly [a.k.a., academic, peer-reviewed] journal. The article may be assigned by the professor, chosen from course readings listed in the syllabus, or you must locate an article on your own, usually with the requirement that you search using a reputable library database, such as, JSTOR or ProQuest . The article chosen is expected to relate to the overall discipline of the course, specific course content, or key concepts discussed in class. In some cases, the purpose of the assignment is to analyze an article that is part of the literature review for a future research project.

Analysis of an article can be assigned to students individually or as part of a small group project. The final product is usually in the form of a short paper [typically 1- 6 double-spaced pages] that addresses key questions the professor uses to guide your analysis or that assesses specific parts of a scholarly research study [e.g., the research problem, methodology, discussion, conclusions or findings]. The analysis paper may be shared on a digital course management platform and/or presented to the class for the purpose of promoting a wider discussion about the topic of the study. Although assigned in any level of undergraduate and graduate coursework in the social and behavioral sciences, professors frequently include this assignment in upper division courses to help students learn how to effectively identify, read, and analyze empirical research within their major.

Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students make the most of Scholarly Articles." Library Management 33 (2012): 525-535.

Benefits of Journal Article Analysis Assignments

Analyzing and synthesizing a scholarly journal article is intended to help students obtain the reading and critical thinking skills needed to develop and write their own research papers. This assignment also supports workplace skills where you could be asked to summarize a report or other type of document and report it, for example, during a staff meeting or for a presentation.

There are two broadly defined ways that analyzing a scholarly journal article supports student learning:

Improve Reading Skills

Conducting research requires an ability to review, evaluate, and synthesize prior research studies. Reading prior research requires an understanding of the academic writing style , the type of epistemological beliefs or practices underpinning the research design, and the specific vocabulary and technical terminology [i.e., jargon] used within a discipline. Reading scholarly articles is important because academic writing is unfamiliar to most students; they have had limited exposure to using peer-reviewed journal articles prior to entering college or students have yet to gain exposure to the specific academic writing style of their disciplinary major. Learning how to read scholarly articles also requires careful and deliberate concentration on how authors use specific language and phrasing to convey their research, the problem it addresses, its relationship to prior research, its significance, its limitations, and how authors connect methods of data gathering to the results so as to develop recommended solutions derived from the overall research process.

Improve Comprehension Skills

In addition to knowing how to read scholarly journals articles, students must learn how to effectively interpret what the scholar(s) are trying to convey. Academic writing can be dense, multi-layered, and non-linear in how information is presented. In addition, scholarly articles contain footnotes or endnotes, references to sources, multiple appendices, and, in some cases, non-textual elements [e.g., graphs, charts] that can break-up the reader’s experience with the narrative flow of the study. Analyzing articles helps students practice comprehending these elements of writing, critiquing the arguments being made, reflecting upon the significance of the research, and how it relates to building new knowledge and understanding or applying new approaches to practice. Comprehending scholarly writing also involves thinking critically about where you fit within the overall dialogue among scholars concerning the research problem, finding possible gaps in the research that require further analysis, or identifying where the author(s) has failed to examine fully any specific elements of the study.

In addition, journal article analysis assignments are used by professors to strengthen discipline-specific information literacy skills, either alone or in relation to other tasks, such as, giving a class presentation or participating in a group project. These benefits can include the ability to:

• Effectively paraphrase text, which leads to a more thorough understanding of the overall study;
• Identify and describe strengths and weaknesses of the study and their implications;
• Relate the article to other course readings and in relation to particular research concepts or ideas discussed during class;
• Think critically about the research and summarize complex ideas contained within;
• Plan, organize, and write an effective inquiry-based paper that investigates a research study, evaluates evidence, expounds on the author’s main ideas, and presents an argument concerning the significance and impact of the research in a clear and concise manner;
• Model the type of source summary and critique you should do for any college-level research paper; and,
• Increase interest and engagement with the research problem of the study as well as with the discipline.

Kershaw, Trina C., Jennifer Fugate, and Aminda J. O'Hare. "Teaching Undergraduates to Understand Published Research through Structured Practice in Identifying Key Research Concepts." Scholarship of Teaching and Learning in Psychology . Advance online publication, 2020; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students make the most of Scholarly Articles." Library Management 33 (2012): 525-535; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946.

Structure and Organization

A journal article analysis paper should be written in paragraph format and include an instruction to the study, your analysis of the research, and a conclusion that provides an overall assessment of the author's work, along with an explanation of what you believe is the study's overall impact and significance. Unless the purpose of the assignment is to examine foundational studies published many years ago, you should select articles that have been published relatively recently [e.g., within the past few years].

Since the research has been completed, reference to the study in your paper should be written in the past tense, with your analysis stated in the present tense [e.g., “The author portrayed access to health care services in rural areas as primarily a problem of having reliable transportation. However, I believe the author is overgeneralizing this issue because...”].

Introduction Section

The first section of a journal analysis paper should describe the topic of the article and highlight the author’s main points. This includes describing the research problem and theoretical framework, the rationale for the research, the methods of data gathering and analysis, the key findings, and the author’s final conclusions and recommendations. The narrative should focus on the act of describing rather than analyzing. Think of the introduction as a more comprehensive and detailed descriptive abstract of the study.

Possible questions to help guide your writing of the introduction section may include:

• Who are the authors and what credentials do they hold that contributes to the validity of the study?
• What was the research problem being investigated?
• What type of research design was used to investigate the research problem?
• What theoretical idea(s) and/or research questions were used to address the problem?
• What was the source of the data or information used as evidence for analysis?
• What methods were applied to investigate this evidence?
• What were the author's overall conclusions and key findings?

Critical Analysis Section

The second section of a journal analysis paper should describe the strengths and weaknesses of the study and analyze its significance and impact. This section is where you shift the narrative from describing to analyzing. Think critically about the research in relation to other course readings, what has been discussed in class, or based on your own life experiences. If you are struggling to identify any weaknesses, explain why you believe this to be true. However, no study is perfect, regardless of how laudable its design may be. Given this, think about the repercussions of the choices made by the author(s) and how you might have conducted the study differently. Examples can include contemplating the choice of what sources were included or excluded in support of examining the research problem, the choice of the method used to analyze the data, or the choice to highlight specific recommended courses of action and/or implications for practice over others. Another strategy is to place yourself within the research study itself by thinking reflectively about what may be missing if you had been a participant in the study or if the recommended courses of action specifically targeted you or your community.

Possible questions to help guide your writing of the analysis section may include:

Introduction

• Did the author clearly state the problem being investigated?
• What was your reaction to and perspective on the research problem?
• Was the study’s objective clearly stated? Did the author clearly explain why the study was necessary?
• How well did the introduction frame the scope of the study?
• Did the introduction conclude with a clear purpose statement?

Literature Review

• Did the literature review lay a foundation for understanding the significance of the research problem?
• Did the literature review provide enough background information to understand the problem in relation to relevant contexts [e.g., historical, economic, social, cultural, etc.].
• Did literature review effectively place the study within the domain of prior research? Is anything missing?
• Was the literature review organized by conceptual categories or did the author simply list and describe sources?
• Did the author accurately explain how the data or information were collected?
• Was the data used sufficient in supporting the study of the research problem?
• Was there another methodological approach that could have been more illuminating?
• Give your overall evaluation of the methods used in this article. How much trust would you put in generating relevant findings?

Results and Discussion

• Were the results clearly presented?
• Did you feel that the results support the theoretical and interpretive claims of the author? Why?
• What did the author(s) do especially well in describing or analyzing their results?
• Was the author's evaluation of the findings clearly stated?
• How well did the discussion of the results relate to what is already known about the research problem?
• Was the discussion of the results free of repetition and redundancies?
• What interpretations did the authors make that you think are in incomplete, unwarranted, or overstated?
• Did the conclusion effectively capture the main points of study?
• Did the conclusion address the research questions posed? Do they seem reasonable?
• Were the author’s conclusions consistent with the evidence and arguments presented?
• Has the author explained how the research added new knowledge or understanding?

Overall Writing Style

• If the article included tables, figures, or other non-textual elements, did they contribute to understanding the study?
• Were ideas developed and related in a logical sequence?
• Were transitions between sections of the article smooth and easy to follow?

Overall Evaluation Section

The final section of a journal analysis paper should bring your thoughts together into a coherent assessment of the value of the research study . This section is where the narrative flow transitions from analyzing specific elements of the article to critically evaluating the overall study. Explain what you view as the significance of the research in relation to the overall course content and any relevant discussions that occurred during class. Think about how the article contributes to understanding the overall research problem, how it fits within existing literature on the topic, how it relates to the course, and what it means to you as a student researcher. In some cases, your professor will also ask you to describe your experiences writing the journal article analysis paper as part of a reflective learning exercise.

Possible questions to help guide your writing of the conclusion and evaluation section may include:

• Was the structure of the article clear and well organized?
• Was the topic of current or enduring interest to you?
• What were the main weaknesses of the article? [this does not refer to limitations stated by the author, but what you believe are potential flaws]
• Was any of the information in the article unclear or ambiguous?
• What did you learn from the research? If nothing stood out to you, explain why.
• Assess the originality of the research. Did you believe it contributed new understanding of the research problem?
• Were you persuaded by the author’s arguments?
• If the author made any final recommendations, will they be impactful if applied to practice?
• In what ways could future research build off of this study?
• What implications does the study have for daily life?
• Was the use of non-textual elements, footnotes or endnotes, and/or appendices helpful in understanding the research?
• What lingering questions do you have after analyzing the article?

NOTE: Avoid using quotes. One of the main purposes of writing an article analysis paper is to learn how to effectively paraphrase and use your own words to summarize a scholarly research study and to explain what the research means to you. Using and citing a direct quote from the article should only be done to help emphasize a key point or to underscore an important concept or idea.

Business: The Article Analysis . Fred Meijer Center for Writing, Grand Valley State University; Bachiochi, Peter et al. "Using Empirical Article Analysis to Assess Research Methods Courses." Teaching of Psychology 38 (2011): 5-9; Brosowsky, Nicholaus P. et al. “Teaching Undergraduate Students to Read Empirical Articles: An Evaluation and Revision of the QALMRI Method.” PsyArXi Preprints , 2020; Holster, Kristin. “Article Evaluation Assignment”. TRAILS: Teaching Resources and Innovations Library for Sociology . Washington DC: American Sociological Association, 2016; Kershaw, Trina C., Jennifer Fugate, and Aminda J. O'Hare. "Teaching Undergraduates to Understand Published Research through Structured Practice in Identifying Key Research Concepts." Scholarship of Teaching and Learning in Psychology . Advance online publication, 2020; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Reviewer's Guide . SAGE Reviewer Gateway, SAGE Journals; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946; Gyuris, Emma, and Laura Castell. "To Tell Them or Show Them? How to Improve Science Students’ Skills of Critical Reading." International Journal of Innovation in Science and Mathematics Education 21 (2013): 70-80; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students Make the Most of Scholarly Articles." Library Management 33 (2012): 525-535.

Writing Tip

Not All Scholarly Journal Articles Can Be Critically Analyzed

There are a variety of articles published in scholarly journals that do not fit within the guidelines of an article analysis assignment. This is because the work cannot be empirically examined or it does not generate new knowledge in a way which can be critically analyzed.

If you are required to locate a research study on your own, avoid selecting these types of journal articles:

• Theoretical essays which discuss concepts, assumptions, and propositions, but report no empirical research;
• Statistical or methodological papers that may analyze data, but the bulk of the work is devoted to refining a new measurement, statistical technique, or modeling procedure;
• Articles that review, analyze, critique, and synthesize prior research, but do not report any original research;
• Brief essays devoted to research methods and findings;
• Articles written by scholars in popular magazines or industry trade journals;
• Academic commentary that discusses research trends or emerging concepts and ideas, but does not contain citations to sources; and
• Pre-print articles that have been posted online, but may undergo further editing and revision by the journal's editorial staff before final publication. An indication that an article is a pre-print is that it has no volume, issue, or page numbers assigned to it.

Journal Analysis Assignment - Myers . Writing@CSU, Colorado State University; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36.

• << Previous: Annotated Bibliography
• Next: Giving an Oral Presentation >>
• Last Updated: Jun 3, 2024 9:44 AM
• URL: https://libguides.usc.edu/writingguide/assignments

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Neurol Res Pract

How to use and assess qualitative research methods

Loraine busetto.

1 Department of Neurology, Heidelberg University Hospital, Im Neuenheimer Feld 400, 69120 Heidelberg, Germany

Wolfgang Wick

2 Clinical Cooperation Unit Neuro-Oncology, German Cancer Research Center, Heidelberg, Germany

Christoph Gumbinger

Associated data.

Not applicable.

This paper aims to provide an overview of the use and assessment of qualitative research methods in the health sciences. Qualitative research can be defined as the study of the nature of phenomena and is especially appropriate for answering questions of why something is (not) observed, assessing complex multi-component interventions, and focussing on intervention improvement. The most common methods of data collection are document study, (non-) participant observations, semi-structured interviews and focus groups. For data analysis, field-notes and audio-recordings are transcribed into protocols and transcripts, and coded using qualitative data management software. Criteria such as checklists, reflexivity, sampling strategies, piloting, co-coding, member-checking and stakeholder involvement can be used to enhance and assess the quality of the research conducted. Using qualitative in addition to quantitative designs will equip us with better tools to address a greater range of research problems, and to fill in blind spots in current neurological research and practice.

The aim of this paper is to provide an overview of qualitative research methods, including hands-on information on how they can be used, reported and assessed. This article is intended for beginning qualitative researchers in the health sciences as well as experienced quantitative researchers who wish to broaden their understanding of qualitative research.

What is qualitative research?

Qualitative research is defined as “the study of the nature of phenomena”, including “their quality, different manifestations, the context in which they appear or the perspectives from which they can be perceived” , but excluding “their range, frequency and place in an objectively determined chain of cause and effect” [ 1 ]. This formal definition can be complemented with a more pragmatic rule of thumb: qualitative research generally includes data in form of words rather than numbers [ 2 ].

Why conduct qualitative research?

Because some research questions cannot be answered using (only) quantitative methods. For example, one Australian study addressed the issue of why patients from Aboriginal communities often present late or not at all to specialist services offered by tertiary care hospitals. Using qualitative interviews with patients and staff, it found one of the most significant access barriers to be transportation problems, including some towns and communities simply not having a bus service to the hospital [ 3 ]. A quantitative study could have measured the number of patients over time or even looked at possible explanatory factors – but only those previously known or suspected to be of relevance. To discover reasons for observed patterns, especially the invisible or surprising ones, qualitative designs are needed.

While qualitative research is common in other fields, it is still relatively underrepresented in health services research. The latter field is more traditionally rooted in the evidence-based-medicine paradigm, as seen in " research that involves testing the effectiveness of various strategies to achieve changes in clinical practice, preferably applying randomised controlled trial study designs (...) " [ 4 ]. This focus on quantitative research and specifically randomised controlled trials (RCT) is visible in the idea of a hierarchy of research evidence which assumes that some research designs are objectively better than others, and that choosing a "lesser" design is only acceptable when the better ones are not practically or ethically feasible [ 5 , 6 ]. Others, however, argue that an objective hierarchy does not exist, and that, instead, the research design and methods should be chosen to fit the specific research question at hand – "questions before methods" [ 2 , 7 – 9 ]. This means that even when an RCT is possible, some research problems require a different design that is better suited to addressing them. Arguing in JAMA, Berwick uses the example of rapid response teams in hospitals, which he describes as " a complex, multicomponent intervention – essentially a process of social change" susceptible to a range of different context factors including leadership or organisation history. According to him, "[in] such complex terrain, the RCT is an impoverished way to learn. Critics who use it as a truth standard in this context are incorrect" [ 8 ] . Instead of limiting oneself to RCTs, Berwick recommends embracing a wider range of methods , including qualitative ones, which for "these specific applications, (...) are not compromises in learning how to improve; they are superior" [ 8 ].

Research problems that can be approached particularly well using qualitative methods include assessing complex multi-component interventions or systems (of change), addressing questions beyond “what works”, towards “what works for whom when, how and why”, and focussing on intervention improvement rather than accreditation [ 7 , 9 – 12 ]. Using qualitative methods can also help shed light on the “softer” side of medical treatment. For example, while quantitative trials can measure the costs and benefits of neuro-oncological treatment in terms of survival rates or adverse effects, qualitative research can help provide a better understanding of patient or caregiver stress, visibility of illness or out-of-pocket expenses.

How to conduct qualitative research?

Given that qualitative research is characterised by flexibility, openness and responsivity to context, the steps of data collection and analysis are not as separate and consecutive as they tend to be in quantitative research [ 13 , 14 ]. As Fossey puts it : “sampling, data collection, analysis and interpretation are related to each other in a cyclical (iterative) manner, rather than following one after another in a stepwise approach” [ 15 ]. The researcher can make educated decisions with regard to the choice of method, how they are implemented, and to which and how many units they are applied [ 13 ]. As shown in Fig.  1 , this can involve several back-and-forth steps between data collection and analysis where new insights and experiences can lead to adaption and expansion of the original plan. Some insights may also necessitate a revision of the research question and/or the research design as a whole. The process ends when saturation is achieved, i.e. when no relevant new information can be found (see also below: sampling and saturation). For reasons of transparency, it is essential for all decisions as well as the underlying reasoning to be well-documented.

Iterative research process

While it is not always explicitly addressed, qualitative methods reflect a different underlying research paradigm than quantitative research (e.g. constructivism or interpretivism as opposed to positivism). The choice of methods can be based on the respective underlying substantive theory or theoretical framework used by the researcher [ 2 ].

Data collection

The methods of qualitative data collection most commonly used in health research are document study, observations, semi-structured interviews and focus groups [ 1 , 14 , 16 , 17 ].

Document study

Document study (also called document analysis) refers to the review by the researcher of written materials [ 14 ]. These can include personal and non-personal documents such as archives, annual reports, guidelines, policy documents, diaries or letters.

Observations

Observations are particularly useful to gain insights into a certain setting and actual behaviour – as opposed to reported behaviour or opinions [ 13 ]. Qualitative observations can be either participant or non-participant in nature. In participant observations, the observer is part of the observed setting, for example a nurse working in an intensive care unit [ 18 ]. In non-participant observations, the observer is “on the outside looking in”, i.e. present in but not part of the situation, trying not to influence the setting by their presence. Observations can be planned (e.g. for 3 h during the day or night shift) or ad hoc (e.g. as soon as a stroke patient arrives at the emergency room). During the observation, the observer takes notes on everything or certain pre-determined parts of what is happening around them, for example focusing on physician-patient interactions or communication between different professional groups. Written notes can be taken during or after the observations, depending on feasibility (which is usually lower during participant observations) and acceptability (e.g. when the observer is perceived to be judging the observed). Afterwards, these field notes are transcribed into observation protocols. If more than one observer was involved, field notes are taken independently, but notes can be consolidated into one protocol after discussions. Advantages of conducting observations include minimising the distance between the researcher and the researched, the potential discovery of topics that the researcher did not realise were relevant and gaining deeper insights into the real-world dimensions of the research problem at hand [ 18 ].

Semi-structured interviews

Hijmans & Kuyper describe qualitative interviews as “an exchange with an informal character, a conversation with a goal” [ 19 ]. Interviews are used to gain insights into a person’s subjective experiences, opinions and motivations – as opposed to facts or behaviours [ 13 ]. Interviews can be distinguished by the degree to which they are structured (i.e. a questionnaire), open (e.g. free conversation or autobiographical interviews) or semi-structured [ 2 , 13 ]. Semi-structured interviews are characterized by open-ended questions and the use of an interview guide (or topic guide/list) in which the broad areas of interest, sometimes including sub-questions, are defined [ 19 ]. The pre-defined topics in the interview guide can be derived from the literature, previous research or a preliminary method of data collection, e.g. document study or observations. The topic list is usually adapted and improved at the start of the data collection process as the interviewer learns more about the field [ 20 ]. Across interviews the focus on the different (blocks of) questions may differ and some questions may be skipped altogether (e.g. if the interviewee is not able or willing to answer the questions or for concerns about the total length of the interview) [ 20 ]. Qualitative interviews are usually not conducted in written format as it impedes on the interactive component of the method [ 20 ]. In comparison to written surveys, qualitative interviews have the advantage of being interactive and allowing for unexpected topics to emerge and to be taken up by the researcher. This can also help overcome a provider or researcher-centred bias often found in written surveys, which by nature, can only measure what is already known or expected to be of relevance to the researcher. Interviews can be audio- or video-taped; but sometimes it is only feasible or acceptable for the interviewer to take written notes [ 14 , 16 , 20 ].

Focus groups

Focus groups are group interviews to explore participants’ expertise and experiences, including explorations of how and why people behave in certain ways [ 1 ]. Focus groups usually consist of 6–8 people and are led by an experienced moderator following a topic guide or “script” [ 21 ]. They can involve an observer who takes note of the non-verbal aspects of the situation, possibly using an observation guide [ 21 ]. Depending on researchers’ and participants’ preferences, the discussions can be audio- or video-taped and transcribed afterwards [ 21 ]. Focus groups are useful for bringing together homogeneous (to a lesser extent heterogeneous) groups of participants with relevant expertise and experience on a given topic on which they can share detailed information [ 21 ]. Focus groups are a relatively easy, fast and inexpensive method to gain access to information on interactions in a given group, i.e. “the sharing and comparing” among participants [ 21 ]. Disadvantages include less control over the process and a lesser extent to which each individual may participate. Moreover, focus group moderators need experience, as do those tasked with the analysis of the resulting data. Focus groups can be less appropriate for discussing sensitive topics that participants might be reluctant to disclose in a group setting [ 13 ]. Moreover, attention must be paid to the emergence of “groupthink” as well as possible power dynamics within the group, e.g. when patients are awed or intimidated by health professionals.

Choosing the “right” method

As explained above, the school of thought underlying qualitative research assumes no objective hierarchy of evidence and methods. This means that each choice of single or combined methods has to be based on the research question that needs to be answered and a critical assessment with regard to whether or to what extent the chosen method can accomplish this – i.e. the “fit” between question and method [ 14 ]. It is necessary for these decisions to be documented when they are being made, and to be critically discussed when reporting methods and results.

Let us assume that our research aim is to examine the (clinical) processes around acute endovascular treatment (EVT), from the patient’s arrival at the emergency room to recanalization, with the aim to identify possible causes for delay and/or other causes for sub-optimal treatment outcome. As a first step, we could conduct a document study of the relevant standard operating procedures (SOPs) for this phase of care – are they up-to-date and in line with current guidelines? Do they contain any mistakes, irregularities or uncertainties that could cause delays or other problems? Regardless of the answers to these questions, the results have to be interpreted based on what they are: a written outline of what care processes in this hospital should look like. If we want to know what they actually look like in practice, we can conduct observations of the processes described in the SOPs. These results can (and should) be analysed in themselves, but also in comparison to the results of the document analysis, especially as regards relevant discrepancies. Do the SOPs outline specific tests for which no equipment can be observed or tasks to be performed by specialized nurses who are not present during the observation? It might also be possible that the written SOP is outdated, but the actual care provided is in line with current best practice. In order to find out why these discrepancies exist, it can be useful to conduct interviews. Are the physicians simply not aware of the SOPs (because their existence is limited to the hospital’s intranet) or do they actively disagree with them or does the infrastructure make it impossible to provide the care as described? Another rationale for adding interviews is that some situations (or all of their possible variations for different patient groups or the day, night or weekend shift) cannot practically or ethically be observed. In this case, it is possible to ask those involved to report on their actions – being aware that this is not the same as the actual observation. A senior physician’s or hospital manager’s description of certain situations might differ from a nurse’s or junior physician’s one, maybe because they intentionally misrepresent facts or maybe because different aspects of the process are visible or important to them. In some cases, it can also be relevant to consider to whom the interviewee is disclosing this information – someone they trust, someone they are otherwise not connected to, or someone they suspect or are aware of being in a potentially “dangerous” power relationship to them. Lastly, a focus group could be conducted with representatives of the relevant professional groups to explore how and why exactly they provide care around EVT. The discussion might reveal discrepancies (between SOPs and actual care or between different physicians) and motivations to the researchers as well as to the focus group members that they might not have been aware of themselves. For the focus group to deliver relevant information, attention has to be paid to its composition and conduct, for example, to make sure that all participants feel safe to disclose sensitive or potentially problematic information or that the discussion is not dominated by (senior) physicians only. The resulting combination of data collection methods is shown in Fig.  2 .

Possible combination of data collection methods

Attributions for icons: “Book” by Serhii Smirnov, “Interview” by Adrien Coquet, FR, “Magnifying Glass” by anggun, ID, “Business communication” by Vectors Market; all from the Noun Project

The combination of multiple data source as described for this example can be referred to as “triangulation”, in which multiple measurements are carried out from different angles to achieve a more comprehensive understanding of the phenomenon under study [ 22 , 23 ].

Data analysis

To analyse the data collected through observations, interviews and focus groups these need to be transcribed into protocols and transcripts (see Fig.  3 ). Interviews and focus groups can be transcribed verbatim , with or without annotations for behaviour (e.g. laughing, crying, pausing) and with or without phonetic transcription of dialects and filler words, depending on what is expected or known to be relevant for the analysis. In the next step, the protocols and transcripts are coded , that is, marked (or tagged, labelled) with one or more short descriptors of the content of a sentence or paragraph [ 2 , 15 , 23 ]. Jansen describes coding as “connecting the raw data with “theoretical” terms” [ 20 ]. In a more practical sense, coding makes raw data sortable. This makes it possible to extract and examine all segments describing, say, a tele-neurology consultation from multiple data sources (e.g. SOPs, emergency room observations, staff and patient interview). In a process of synthesis and abstraction, the codes are then grouped, summarised and/or categorised [ 15 , 20 ]. The end product of the coding or analysis process is a descriptive theory of the behavioural pattern under investigation [ 20 ]. The coding process is performed using qualitative data management software, the most common ones being InVivo, MaxQDA and Atlas.ti. It should be noted that these are data management tools which support the analysis performed by the researcher(s) [ 14 ].

From data collection to data analysis

Attributions for icons: see Fig. ​ Fig.2, 2 , also “Speech to text” by Trevor Dsouza, “Field Notes” by Mike O’Brien, US, “Voice Record” by ProSymbols, US, “Inspection” by Made, AU, and “Cloud” by Graphic Tigers; all from the Noun Project

How to report qualitative research?

Protocols of qualitative research can be published separately and in advance of the study results. However, the aim is not the same as in RCT protocols, i.e. to pre-define and set in stone the research questions and primary or secondary endpoints. Rather, it is a way to describe the research methods in detail, which might not be possible in the results paper given journals’ word limits. Qualitative research papers are usually longer than their quantitative counterparts to allow for deep understanding and so-called “thick description”. In the methods section, the focus is on transparency of the methods used, including why, how and by whom they were implemented in the specific study setting, so as to enable a discussion of whether and how this may have influenced data collection, analysis and interpretation. The results section usually starts with a paragraph outlining the main findings, followed by more detailed descriptions of, for example, the commonalities, discrepancies or exceptions per category [ 20 ]. Here it is important to support main findings by relevant quotations, which may add information, context, emphasis or real-life examples [ 20 , 23 ]. It is subject to debate in the field whether it is relevant to state the exact number or percentage of respondents supporting a certain statement (e.g. “Five interviewees expressed negative feelings towards XYZ”) [ 21 ].

How to combine qualitative with quantitative research?

Qualitative methods can be combined with other methods in multi- or mixed methods designs, which “[employ] two or more different methods [ …] within the same study or research program rather than confining the research to one single method” [ 24 ]. Reasons for combining methods can be diverse, including triangulation for corroboration of findings, complementarity for illustration and clarification of results, expansion to extend the breadth and range of the study, explanation of (unexpected) results generated with one method with the help of another, or offsetting the weakness of one method with the strength of another [ 1 , 17 , 24 – 26 ]. The resulting designs can be classified according to when, why and how the different quantitative and/or qualitative data strands are combined. The three most common types of mixed method designs are the convergent parallel design , the explanatory sequential design and the exploratory sequential design. The designs with examples are shown in Fig.  4 .

Three common mixed methods designs

In the convergent parallel design, a qualitative study is conducted in parallel to and independently of a quantitative study, and the results of both studies are compared and combined at the stage of interpretation of results. Using the above example of EVT provision, this could entail setting up a quantitative EVT registry to measure process times and patient outcomes in parallel to conducting the qualitative research outlined above, and then comparing results. Amongst other things, this would make it possible to assess whether interview respondents’ subjective impressions of patients receiving good care match modified Rankin Scores at follow-up, or whether observed delays in care provision are exceptions or the rule when compared to door-to-needle times as documented in the registry. In the explanatory sequential design, a quantitative study is carried out first, followed by a qualitative study to help explain the results from the quantitative study. This would be an appropriate design if the registry alone had revealed relevant delays in door-to-needle times and the qualitative study would be used to understand where and why these occurred, and how they could be improved. In the exploratory design, the qualitative study is carried out first and its results help informing and building the quantitative study in the next step [ 26 ]. If the qualitative study around EVT provision had shown a high level of dissatisfaction among the staff members involved, a quantitative questionnaire investigating staff satisfaction could be set up in the next step, informed by the qualitative study on which topics dissatisfaction had been expressed. Amongst other things, the questionnaire design would make it possible to widen the reach of the research to more respondents from different (types of) hospitals, regions, countries or settings, and to conduct sub-group analyses for different professional groups.

How to assess qualitative research?

A variety of assessment criteria and lists have been developed for qualitative research, ranging in their focus and comprehensiveness [ 14 , 17 , 27 ]. However, none of these has been elevated to the “gold standard” in the field. In the following, we therefore focus on a set of commonly used assessment criteria that, from a practical standpoint, a researcher can look for when assessing a qualitative research report or paper.

Assessors should check the authors’ use of and adherence to the relevant reporting checklists (e.g. Standards for Reporting Qualitative Research (SRQR)) to make sure all items that are relevant for this type of research are addressed [ 23 , 28 ]. Discussions of quantitative measures in addition to or instead of these qualitative measures can be a sign of lower quality of the research (paper). Providing and adhering to a checklist for qualitative research contributes to an important quality criterion for qualitative research, namely transparency [ 15 , 17 , 23 ].

Reflexivity

While methodological transparency and complete reporting is relevant for all types of research, some additional criteria must be taken into account for qualitative research. This includes what is called reflexivity, i.e. sensitivity to the relationship between the researcher and the researched, including how contact was established and maintained, or the background and experience of the researcher(s) involved in data collection and analysis. Depending on the research question and population to be researched this can be limited to professional experience, but it may also include gender, age or ethnicity [ 17 , 27 ]. These details are relevant because in qualitative research, as opposed to quantitative research, the researcher as a person cannot be isolated from the research process [ 23 ]. It may influence the conversation when an interviewed patient speaks to an interviewer who is a physician, or when an interviewee is asked to discuss a gynaecological procedure with a male interviewer, and therefore the reader must be made aware of these details [ 19 ].

Sampling and saturation

The aim of qualitative sampling is for all variants of the objects of observation that are deemed relevant for the study to be present in the sample “ to see the issue and its meanings from as many angles as possible” [ 1 , 16 , 19 , 20 , 27 ] , and to ensure “information-richness [ 15 ]. An iterative sampling approach is advised, in which data collection (e.g. five interviews) is followed by data analysis, followed by more data collection to find variants that are lacking in the current sample. This process continues until no new (relevant) information can be found and further sampling becomes redundant – which is called saturation [ 1 , 15 ] . In other words: qualitative data collection finds its end point not a priori , but when the research team determines that saturation has been reached [ 29 , 30 ].

This is also the reason why most qualitative studies use deliberate instead of random sampling strategies. This is generally referred to as “ purposive sampling” , in which researchers pre-define which types of participants or cases they need to include so as to cover all variations that are expected to be of relevance, based on the literature, previous experience or theory (i.e. theoretical sampling) [ 14 , 20 ]. Other types of purposive sampling include (but are not limited to) maximum variation sampling, critical case sampling or extreme or deviant case sampling [ 2 ]. In the above EVT example, a purposive sample could include all relevant professional groups and/or all relevant stakeholders (patients, relatives) and/or all relevant times of observation (day, night and weekend shift).

Assessors of qualitative research should check whether the considerations underlying the sampling strategy were sound and whether or how researchers tried to adapt and improve their strategies in stepwise or cyclical approaches between data collection and analysis to achieve saturation [ 14 ].

Good qualitative research is iterative in nature, i.e. it goes back and forth between data collection and analysis, revising and improving the approach where necessary. One example of this are pilot interviews, where different aspects of the interview (especially the interview guide, but also, for example, the site of the interview or whether the interview can be audio-recorded) are tested with a small number of respondents, evaluated and revised [ 19 ]. In doing so, the interviewer learns which wording or types of questions work best, or which is the best length of an interview with patients who have trouble concentrating for an extended time. Of course, the same reasoning applies to observations or focus groups which can also be piloted.

Ideally, coding should be performed by at least two researchers, especially at the beginning of the coding process when a common approach must be defined, including the establishment of a useful coding list (or tree), and when a common meaning of individual codes must be established [ 23 ]. An initial sub-set or all transcripts can be coded independently by the coders and then compared and consolidated after regular discussions in the research team. This is to make sure that codes are applied consistently to the research data.

Member checking

Member checking, also called respondent validation , refers to the practice of checking back with study respondents to see if the research is in line with their views [ 14 , 27 ]. This can happen after data collection or analysis or when first results are available [ 23 ]. For example, interviewees can be provided with (summaries of) their transcripts and asked whether they believe this to be a complete representation of their views or whether they would like to clarify or elaborate on their responses [ 17 ]. Respondents’ feedback on these issues then becomes part of the data collection and analysis [ 27 ].

Stakeholder involvement

In those niches where qualitative approaches have been able to evolve and grow, a new trend has seen the inclusion of patients and their representatives not only as study participants (i.e. “members”, see above) but as consultants to and active participants in the broader research process [ 31 – 33 ]. The underlying assumption is that patients and other stakeholders hold unique perspectives and experiences that add value beyond their own single story, making the research more relevant and beneficial to researchers, study participants and (future) patients alike [ 34 , 35 ]. Using the example of patients on or nearing dialysis, a recent scoping review found that 80% of clinical research did not address the top 10 research priorities identified by patients and caregivers [ 32 , 36 ]. In this sense, the involvement of the relevant stakeholders, especially patients and relatives, is increasingly being seen as a quality indicator in and of itself.

How not to assess qualitative research

The above overview does not include certain items that are routine in assessments of quantitative research. What follows is a non-exhaustive, non-representative, experience-based list of the quantitative criteria often applied to the assessment of qualitative research, as well as an explanation of the limited usefulness of these endeavours.

Protocol adherence

Given the openness and flexibility of qualitative research, it should not be assessed by how well it adheres to pre-determined and fixed strategies – in other words: its rigidity. Instead, the assessor should look for signs of adaptation and refinement based on lessons learned from earlier steps in the research process.

Sample size

For the reasons explained above, qualitative research does not require specific sample sizes, nor does it require that the sample size be determined a priori [ 1 , 14 , 27 , 37 – 39 ]. Sample size can only be a useful quality indicator when related to the research purpose, the chosen methodology and the composition of the sample, i.e. who was included and why.

Randomisation

While some authors argue that randomisation can be used in qualitative research, this is not commonly the case, as neither its feasibility nor its necessity or usefulness has been convincingly established for qualitative research [ 13 , 27 ]. Relevant disadvantages include the negative impact of a too large sample size as well as the possibility (or probability) of selecting “ quiet, uncooperative or inarticulate individuals ” [ 17 ]. Qualitative studies do not use control groups, either.

Interrater reliability, variability and other “objectivity checks”

The concept of “interrater reliability” is sometimes used in qualitative research to assess to which extent the coding approach overlaps between the two co-coders. However, it is not clear what this measure tells us about the quality of the analysis [ 23 ]. This means that these scores can be included in qualitative research reports, preferably with some additional information on what the score means for the analysis, but it is not a requirement. Relatedly, it is not relevant for the quality or “objectivity” of qualitative research to separate those who recruited the study participants and collected and analysed the data. Experiences even show that it might be better to have the same person or team perform all of these tasks [ 20 ]. First, when researchers introduce themselves during recruitment this can enhance trust when the interview takes place days or weeks later with the same researcher. Second, when the audio-recording is transcribed for analysis, the researcher conducting the interviews will usually remember the interviewee and the specific interview situation during data analysis. This might be helpful in providing additional context information for interpretation of data, e.g. on whether something might have been meant as a joke [ 18 ].

Not being quantitative research

Being qualitative research instead of quantitative research should not be used as an assessment criterion if it is used irrespectively of the research problem at hand. Similarly, qualitative research should not be required to be combined with quantitative research per se – unless mixed methods research is judged as inherently better than single-method research. In this case, the same criterion should be applied for quantitative studies without a qualitative component.

The main take-away points of this paper are summarised in Table ​ Table1. 1 . We aimed to show that, if conducted well, qualitative research can answer specific research questions that cannot to be adequately answered using (only) quantitative designs. Seeing qualitative and quantitative methods as equal will help us become more aware and critical of the “fit” between the research problem and our chosen methods: I can conduct an RCT to determine the reasons for transportation delays of acute stroke patients – but should I? It also provides us with a greater range of tools to tackle a greater range of research problems more appropriately and successfully, filling in the blind spots on one half of the methodological spectrum to better address the whole complexity of neurological research and practice.

Take-away-points

Acknowledgements

Abbreviations.

Authors’ contributions

LB drafted the manuscript; WW and CG revised the manuscript; all authors approved the final versions.

no external funding.

Availability of data and materials

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Research Analysis & Evaluation

International Double Blind Peer Reviewed, Refereed, Multilingual,Multidisciplinary & Indexed- Monthly Research Journal

• ISSN(E) : 2320-5482 RNI : RAJBIL2009/30097
• Impact Factor : 6.376 (SJIF)

Hon'ble Professor & Research Scholars

RESEARCH ANALYSIS AND EVALUATION is an International Research journal  waiting for your Research Paper publication.This is monthly,Referred, interdiciplinery and multilingula (English,Hindi,Marathi & Gujarati) Research journal so now you can send your Research paper for Publication .Please send your Research paper online to click here SUBMIT ONLINE PAPER or can send from above menu For more details Click here & down load MEMBERSHIP AND COPYRIGHT AGREEMENT FORM  or above MENU from DOWNLOAD you can find this form,and follow the instuctions mention in Membership form. We hope that all Author will write there Research Papers according to Research Mathodology.

Kabir-By Professor.K

Kal sarp dosh, advertisement a.

ASK QUESTION TO THE LIFE..

DEFINITION OF SWEET LIFE

ALL ANSWERS OF YOUR QUESTION

RELATED BLOGS

Helpful links.

• ugc journal

A-215, Moti Nagar,Street No-7 , Queens Road , Jaipur-302021, Rajasthan.

• [email protected]

© Copyright 2016. Designed by: Corporate1 Software

• Understanding Evaluation Methodologies: M&E Methods and Techniques for Assessing Performance and Impact
• Learning Center

This article provides an overview and comparison of the different types of evaluation methodologies used to assess the performance, effectiveness, quality, or impact of services, programs, and policies. There are several methodologies both qualitative and quantitative, including surveys, interviews, observations, case studies, focus groups, and more…In this essay, we will discuss the most commonly used qualitative and quantitative evaluation methodologies in the M&E field.

Table of Contents

• Introduction to Evaluation Methodologies: Definition and Importance
• Types of Evaluation Methodologies: Overview and Comparison
• Program Evaluation methodologies
• Qualitative Methodologies in Monitoring and Evaluation (M&E)
• Quantitative Methodologies in Monitoring and Evaluation (M&E)
• What are the M&E Methods?
• Difference Between Evaluation Methodologies and M&E Methods
• Choosing the Right Evaluation Methodology: Factors and Criteria
• Our Conclusion on Evaluation Methodologies

1. Introduction to Evaluation Methodologies: Definition and Importance

Evaluation methodologies are the methods and techniques used to measure the performance, effectiveness, quality, or impact of various interventions, services, programs, and policies. Evaluation is essential for decision-making, improvement, and innovation, as it helps stakeholders identify strengths, weaknesses, opportunities, and threats and make informed decisions to improve the effectiveness and efficiency of their operations.

Evaluation methodologies can be used in various fields and industries, such as healthcare, education, business, social services, and public policy. The choice of evaluation methodology depends on the specific goals of the evaluation, the type and level of data required, and the resources available for conducting the evaluation.

The importance of evaluation methodologies lies in their ability to provide evidence-based insights into the performance and impact of the subject being evaluated. This information can be used to guide decision-making, policy development, program improvement, and innovation. By using evaluation methodologies, stakeholders can assess the effectiveness of their operations and make data-driven decisions to improve their outcomes.

Overall, understanding evaluation methodologies is crucial for individuals and organizations seeking to enhance their performance, effectiveness, and impact. By selecting the appropriate evaluation methodology and conducting a thorough evaluation, stakeholders can gain valuable insights and make informed decisions to improve their operations and achieve their goals.

2. Types of Evaluation Methodologies: Overview and Comparison

Evaluation methodologies can be categorized into two main types based on the type of data they collect: qualitative and quantitative. Qualitative methodologies collect non-numerical data, such as words, images, or observations, while quantitative methodologies collect numerical data that can be analyzed statistically. Here is an overview and comparison of the main differences between qualitative and quantitative evaluation methodologies:

Qualitative Evaluation Methodologies:

• Collect non-numerical data, such as words, images, or observations.
• Focus on exploring complex phenomena, such as attitudes, perceptions, and behaviors, and understanding the meaning and context behind them.
• Use techniques such as interviews, observations, case studies, and focus groups to collect data.
• Emphasize the subjective nature of the data and the importance of the researcher’s interpretation and analysis.
• Provide rich and detailed insights into people’s experiences and perspectives.
• Limitations include potential bias from the researcher, limited generalizability of findings, and challenges in analyzing and synthesizing the data.

Quantitative Evaluation Methodologies:

• Collect numerical data that can be analyzed statistically.
• Focus on measuring specific variables and relationships between them, such as the effectiveness of an intervention or the correlation between two factors.
• Use techniques such as surveys and experimental designs to collect data.
• Emphasize the objectivity of the data and the importance of minimizing bias and variability.
• Provide precise and measurable data that can be compared and analyzed statistically.
• Limitations include potential oversimplification of complex phenomena, limited contextual information, and challenges in collecting and analyzing data.

Choosing between qualitative and quantitative evaluation methodologies depends on the specific goals of the evaluation, the type and level of data required, and the resources available for conducting the evaluation. Some evaluations may use a mixed-methods approach that combines both qualitative and quantitative data collection and analysis techniques to provide a more comprehensive understanding of the subject being evaluated.

3. Program evaluation methodologies

Program evaluation methodologies encompass a diverse set of approaches and techniques used to assess the effectiveness, efficiency, and impact of programs and interventions. These methodologies provide systematic frameworks for collecting, analyzing, and interpreting data to determine the extent to which program objectives are being met and to identify areas for improvement. Common program evaluation methodologies include quantitative methods such as experimental designs, quasi-experimental designs, and surveys, as well as qualitative approaches like interviews, focus groups, and case studies.

Each methodology offers unique advantages and limitations depending on the nature of the program being evaluated, the available resources, and the research questions at hand. By employing rigorous program evaluation methodologies, organizations can make informed decisions, enhance program effectiveness, and maximize the use of resources to achieve desired outcomes.

Catch HR’s eye instantly?

• Resume Review
• Resume Writing
• Resume Optimization

Premier global development resume service since 2012

Stand Out with a Pro Resume

4. Qualitative Methodologies in Monitoring and Evaluation (M&E)

Qualitative methodologies are increasingly being used in monitoring and evaluation (M&E) to provide a more comprehensive understanding of the impact and effectiveness of programs and interventions. Qualitative methodologies can help to explore the underlying reasons and contexts that contribute to program outcomes and identify areas for improvement. Here are some common qualitative methodologies used in M&E:

Interviews involve one-on-one or group discussions with stakeholders to collect data on their experiences, perspectives, and perceptions. Interviews can provide rich and detailed data on the effectiveness of a program, the factors that contribute to its success or failure, and the ways in which it can be improved.

Observations

Observations involve the systematic and objective recording of behaviors and interactions of stakeholders in a natural setting. Observations can help to identify patterns of behavior, the effectiveness of program interventions, and the ways in which they can be improved.

Document review

Document review involves the analysis of program documents, such as reports, policies, and procedures, to understand the program context, design, and implementation. Document review can help to identify gaps in program design or implementation and suggest ways in which they can be improved.

Participatory Rural Appraisal (PRA)

PRA is a participatory approach that involves working with communities to identify and analyze their own problems and challenges. It involves using participatory techniques such as mapping, focus group discussions, and transect walks to collect data on community perspectives, experiences, and priorities. PRA can help ensure that the evaluation is community-driven and culturally appropriate, and can provide valuable insights into the social and cultural factors that influence program outcomes.

Key Informant Interviews

Key informant interviews are in-depth, open-ended interviews with individuals who have expert knowledge or experience related to the program or issue being evaluated. Key informants can include program staff, community leaders, or other stakeholders. These interviews can provide valuable insights into program implementation and effectiveness, and can help identify areas for improvement.

Ethnography

Ethnography is a qualitative method that involves observing and immersing oneself in a community or culture to understand their perspectives, values, and behaviors. Ethnographic methods can include participant observation, interviews, and document analysis, among others. Ethnography can provide a more holistic understanding of program outcomes and impacts, as well as the broader social context in which the program operates.

Focus Group Discussions

Focus group discussions involve bringing together a small group of individuals to discuss a specific topic or issue related to the program. Focus group discussions can be used to gather qualitative data on program implementation, participant experiences, and program outcomes. They can also provide insights into the diversity of perspectives within a community or stakeholder group .

Photovoice is a qualitative method that involves using photography as a tool for community empowerment and self-expression. Participants are given cameras and asked to take photos that represent their experiences or perspectives on a program or issue. These photos can then be used to facilitate group discussions and generate qualitative data on program outcomes and impacts.

Case Studies

Case studies involve gathering detailed qualitative data through interviews, document analysis, and observation, and can provide a more in-depth understanding of a specific program component. They can be used to explore the experiences and perspectives of program participants or stakeholders and can provide insights into program outcomes and impacts.

Qualitative methodologies in M&E are useful for identifying complex and context-dependent factors that contribute to program outcomes, and for exploring stakeholder perspectives and experiences. Qualitative methodologies can provide valuable insights into the ways in which programs can be improved and can complement quantitative methodologies in providing a comprehensive understanding of program impact and effectiveness

5. Quantitative Methodologies in Monitoring and Evaluation (M&E)

Quantitative methodologies are commonly used in monitoring and evaluation (M&E) to measure program outcomes and impact in a systematic and objective manner. Quantitative methodologies involve collecting numerical data that can be analyzed statistically to provide insights into program effectiveness, efficiency, and impact. Here are some common quantitative methodologies used in M&E:

Surveys involve collecting data from a large number of individuals using standardized questionnaires or surveys. Surveys can provide quantitative data on people’s attitudes, opinions, behaviors, and experiences, and can help to measure program outcomes and impact.

Baseline and Endline Surveys

Baseline and endline surveys are quantitative surveys conducted at the beginning and end of a program to measure changes in knowledge, attitudes, behaviors, or other outcomes. These surveys can provide a snapshot of program impact and allow for comparisons between pre- and post-program data.

Randomized Controlled Trials (RCTs)

RCTs are a rigorous quantitative evaluation method that involve randomly assigning participants to a treatment group (receiving the program) and a control group (not receiving the program), and comparing outcomes between the two groups. RCTs are often used to assess the impact of a program.

Cost-Benefit Analysis

Cost-benefit analysis is a quantitative method used to assess the economic efficiency of a program or intervention. It involves comparing the costs of the program with the benefits or outcomes generated, and can help determine whether a program is cost-effective or not.

Performance Indicators

Performance indicator s are quantitative measures used to track progress toward program goals and objectives. These indicators can be used to assess program effectiveness, efficiency, and impact, and can provide regular feedback on program performance.

Statistical Analysis

Statistical analysis involves using quantitative data and statistical method s to analyze data gathered from various evaluation methods, such as surveys or observations. Statistical analysis can provide a more rigorous assessment of program outcomes and impacts and help identify patterns or relationships between variables.

Experimental designs

Experimental designs involve manipulating one or more variables and measuring the effects of the manipulation on the outcome of interest. Experimental designs are useful for establishing cause-and-effect relationships between variables, and can help to measure the effectiveness of program interventions.

Quantitative methodologies in M&E are useful for providing objective and measurable data on program outcomes and impact, and for identifying patterns and trends in program performance. Quantitative methodologies can provide valuable insights into the effectiveness, efficiency, and impact of programs, and can complement qualitative methodologies in providing a comprehensive understanding of program performance.

6. What are the M&E Methods?

Monitoring and Evaluation (M&E) methods encompass the tools, techniques, and processes used to assess the performance of projects, programs, or policies.

These methods are essential in determining whether the objectives are being met, understanding the impact of interventions, and guiding decision-making for future improvements. M&E methods fall into two broad categories: qualitative and quantitative, often used in combination for a comprehensive evaluation.

7. Choosing the Right Evaluation Methodology: Factors and Criteria

Choosing the right evaluation methodology is essential for conducting an effective and meaningful evaluation. Here are some factors and criteria to consider when selecting an appropriate evaluation methodology:

• Evaluation goals and objectives: The evaluation goals and objectives should guide the selection of an appropriate methodology. For example, if the goal is to explore stakeholders’ perspectives and experiences, qualitative methodologies such as interviews or focus groups may be more appropriate. If the goal is to measure program outcomes and impact, quantitative methodologies such as surveys or experimental designs may be more appropriate.
• Type of data required: The type of data required for the evaluation should also guide the selection of the methodology. Qualitative methodologies collect non-numerical data, such as words, images, or observations, while quantitative methodologies collect numerical data that can be analyzed statistically. The type of data required will depend on the evaluation goals and objectives.
• Resources available: The resources available, such as time, budget, and expertise, can also influence the selection of an appropriate methodology. Some methodologies may require more resources, such as specialized expertise or equipment, while others may be more cost-effective and easier to implement.
• Accessibility of the subject being evaluated: The accessibility of the subject being evaluated, such as the availability of stakeholders or data, can also influence the selection of an appropriate methodology. For example, if stakeholders are geographically dispersed, remote data collection methods such as online surveys or video conferencing may be more appropriate.
• Ethical considerations: Ethical considerations, such as ensuring the privacy and confidentiality of stakeholders, should also be taken into account when selecting an appropriate methodology. Some methodologies, such as interviews or focus groups, may require more attention to ethical considerations than others.

Overall, choosing the right evaluation methodology depends on a variety of factors and criteria, including the evaluation goals and objectives, the type of data required, the resources available, the accessibility of the subject being evaluated, and ethical considerations. Selecting an appropriate methodology can ensure that the evaluation is effective, meaningful, and provides valuable insights into program performance and impact.

8. Our Conclusion on Evaluation Methodologies

It’s worth noting that many evaluation methodologies use a combination of quantitative and qualitative methods to provide a more comprehensive understanding of program outcomes and impacts. Both qualitative and quantitative methodologies are essential in providing insights into program performance and effectiveness.

Qualitative methodologies focus on gathering data on the experiences, perspectives, and attitudes of individuals or communities involved in a program, providing a deeper understanding of the social and cultural factors that influence program outcomes. In contrast, quantitative methodologies focus on collecting numerical data on program performance and impact, providing more rigorous evidence of program effectiveness and efficiency.

Each methodology has its strengths and limitations, and a combination of both qualitative and quantitative approaches is often the most effective in providing a comprehensive understanding of program outcomes and impact. When designing an M&E plan, it is crucial to consider the program’s objectives, context, and stakeholders to select the most appropriate methodologies.

Overall, effective M&E practices require a systematic and continuous approach to data collection, analysis, and reporting. With the right combination of qualitative and quantitative methodologies, M&E can provide valuable insights into program performance, progress, and impact, enabling informed decision-making and resource allocation, ultimately leading to more successful and impactful programs.

Munir Barnaba

Thanks for your help its of high value, much appreciated

Very informative. Thank you

Chokri HAMOUDA

I am grateful for this article, which offers valuable insights and serves as an excellent educational resource. My thanks go to the author.

Leave a Comment Cancel Reply

You must be logged in to post a comment.

How strong is my Resume?

Only 2% of resumes land interviews.

Land a better, higher-paying career

Jobs for You

Request for proposal (rfp) for global labor evaluation.

• Washington, DC, USA (Remote)
• Solidarity Center

Program Analyst – Cuba

• United States

Human Resources Consultant

Specialist, human resources, program advisor, mel.

• United States (Remote)

Communications and Analytics Advisor

Power generation / energy public private partnership (ppp) specialist.

• Côte d'Ivoire

Subject Matter Expert (Media Literacy)

• North Macedonia

Evaluation Specialist

Senior associate, human resources, team leader, college of education: open-rank, evaluation/social research methods — educational psychology.

• Champaign, IL, USA
• University of Illinois at Urbana-Champaign

Deputy Director – Operations and Finance

Energy/environment senior advisor, climate finance specialist, services you might be interested in, useful guides ....

How to Create a Strong Resume

Monitoring And Evaluation Specialist Resume

Resume Length for the International Development Sector

Types of Evaluation

Monitoring, Evaluation, Accountability, and Learning (MEAL)

LAND A JOB REFERRAL IN 2 WEEKS (NO ONLINE APPS!)

Sign Up & To Get My Free Referral Toolkit Now:

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Journal Proposal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The performance and qualitative evaluation of scientific work at research universities: a focus on the types of university and research.

1. Introduction

2. materials and methods, 3. literature review, 4.1. description of the research object and university research data analysis, 4.2. survey result analysis, 5. discussion, 6. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest, appendix a. survey form for indicators for assessing the quality of the scientific research, appendix b. university research processes.

Click here to enlarge figure

• Etzkowitz, H. The Triple Helix University—Industry—Government Innovation in Action ; Routledge: New York, NY, USA, 2008; p. 225. ISBN 978-0415964500. [ Google Scholar ]
• Tang, H.H. The strategic role of world-class universities in regional innovation system: China’s Greater Bay Area and Hong Kong’s academic profession. Asian Educ. Dev. Stud. 2020 , 11 , 7–22. [ Google Scholar ] [ CrossRef ]
• National Research Council, USA. Research Universities and the Future of America: Ten Breakthrough Actions Vital to Our Nation’s Prosperity and Security (Report) ; National Research Council: Washington, DC, USA, 2012; p. 225. ISBN 978-0-309-25639-1.
• Powell, J.J.W.; Dusdal, J. The European Center of Science Productivity: Research Universities and Institutes in France, Germany, and the United Kingdom. Century Sci. Int. Perspect. Educ. Soc. 2017 , 33 , 55–83. [ Google Scholar ] [ CrossRef ]
• Intarakumnerd, P.; Goto, A. Role of public research institutes in national innovation systems in industrialized countries: The cases of Fraunhofer, NIST, CSIRO, AIST, and ITRI. Res. Policy 2018 , 47 , 1309–1320. [ Google Scholar ] [ CrossRef ]
• Vlasova, V.V.; Gokhberg, L.M.; Ditkovsky, K.A.; Kotsemir, M.N.; Kuznetsova, I.A.; Martynova, S.V.; Nesterenko, A.V.; Pakhomov, S.I.; Polyakova, V.V.; Ratay, T.V.; et al. Science Indicators: 2023: Statistical Collection ; National Research University Higher School of Economics: Moscow, Russia, 2023; p. 416. ISBN 978-5-7598-2765-8. (In Russian) [ Google Scholar ]
• Textor, C. Breakdown of R&D spending in China 2017–2022, by Entity. Available online: https://www.statista.com/statistics/1465556/research-and-development-expenditure-in-china-distribution-by-entity/ (accessed on 12 August 2024).
• Chen, K.; Zhang, C.; Feng, Z.; Zhang, Y.; Ning, L. Technology transfer systems and modes of national research institutes: Evidence from the Chinese Academy of Sciences. Res. Policy 2022 , 51 , 104471. [ Google Scholar ] [ CrossRef ]
• Vasilev, Y.; Vasileva, P.; Batova, O.; Tsvetkova, A. Assessment of Factors Influencing Educational Effectiveness in Higher Educational Institutions. Sustainability 2024 , 16 , 4886. [ Google Scholar ] [ CrossRef ]
• Ilyushin, Y.V.; Pervukhin, D.A.; Afanaseva, O.V. Application of the theory of systems with distributed parameters for mineral complex facilities management. ARPN J. Eng. Appl. Sci. 2019 , 14 , 3852–3864. [ Google Scholar ]
• Raupov, I.; Burkhanov, R.; Lutfullin, A.; Maksyutin, A.; Lebedev, A.; Safiullina, E. Experience in the Application of Hydrocarbon Optical Studies in Oil Field Development. Energies 2022 , 15 , 3626. [ Google Scholar ] [ CrossRef ]
• Rudnik, S.N.; Afanasev, V.G.; Samylovskaya, E.A. 250 years in the service of the Fatherland: Empress Catherine II Saint Petersburg Mining university in facts and figures. J. Min. Inst. 2023 , 263 , 810–830. [ Google Scholar ]
• Olcay, G.F.; Bulu, M. Is measuring the knowledge creation of universities possible? A review of university rankings. Technol. Forecast. Soc. Chang. 2017 , 123 , 153–160. [ Google Scholar ] [ CrossRef ]
• Lapinskas, A.; Makhova, L.; Zhidikov, V. Responsible resource wealth management in ensuring inclusive growth [Odpowiedzialne zarzdzanie zasobami w zapewnieniu wzrostu wczajcego]. Pol. J. Manag. Stud. 2021 , 23 , 288–304. [ Google Scholar ] [ CrossRef ]
• Peris-Ortiz, M.; García-Hurtado, D.; Román, A.P. Measuring knowledge exploration and exploitation in universities and the relationship with global ranking indicators. Eur. Res. Manag. Bus. Econ. 2023 , 29 , 100212. [ Google Scholar ] [ CrossRef ]
• Ponomariov, B.L.; Boardman, P.C. Influencing scientists’ collaboration and productivity patterns through new institutions: University research centers and scientific and technical human capital. Res. Policy 2010 , 39 , 613–624. [ Google Scholar ] [ CrossRef ]
• Isaeva, N.V.; Borisova, L.V. Comparative analysis of national policies for developing research universities’ campuses. Univ. Manag. Pract. Anal. 2013 , 6 , 74–87. (In Russian) [ Google Scholar ]
• Ponomarenko, T.V.; Marinina, O.A.; Nevskaya, M.A. Innovative learning methods in technical universities: The possibility of forming interdisciplinary competencies. Espacios 2019 , 40 , 1–10. Available online: http://www.revistaespacios.com/a19v40n41/19404116.html (accessed on 10 September 2024).
• Zhang, H.; Patton, D.; Kenney, M. Building global-class universities: Assessing the impact of the 985 Project. Res. Policy 2013 , 42 , 765–775. [ Google Scholar ] [ CrossRef ]
• Shang, J.; Zeng, M.; Zhang, G. Investigating the mentorship effect on the academic success of young scientists: An empirical study of the 985 project universities of China. J. Informetr. 2022 , 16 , 101285. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Pan, Y.; Liu, H.; Wu, X.; Deng, G. Efficiency analysis of Chinese universities with shared inputs: An aggregated two-stage network DEA approach. Socio-Econ. Plan. Sci. 2023 , 90 , 101728. [ Google Scholar ] [ CrossRef ]
• Mindeli, L.E. Financial Support for the Development of the Scientific and Technological Sphere ; Mindeli, L.E., Chernykh, S.I., Frolova, N.D., Todosiychuk, A.V., Fetisov, V.P., Eds.; Institute for Problems of Science Development of the Russian Academy of Sciences (IPRAN): Moscow, Russia, 2018; p. 215. ISBN 978-5-91294-125-2. (In Russian) [ Google Scholar ]
• Guba, K.S.; Slovogorodsky, N.A. “Publish or Perish” in Russian social sciences: Patterns of co-authorship in “predatory” and “pure” journals. Issues Educ. Educ. Stud. Mosc. 2022 , 4 , 80–106. (In Russian) [ Google Scholar ] [ CrossRef ]
• Slepykh, V.; Lovakov, A.; Yudkevich, M. Academic career after defending a PhD thesis on the example of four branches of Russian science. Issues Educ. Educ. Stud. Mosc. 2022 , 4 , 260–297. (In Russian) [ Google Scholar ] [ CrossRef ]
• Mohrman, K.; Ma, W.; Baker, D. The Research University in Transition: The Emerging Global Model. High. Educ. Policy 2008 , 21 , 5–27. [ Google Scholar ] [ CrossRef ]
• Altbach, P.G. The Road to Academic Excellence: The Making of World-Class Research Universities ; Altbach, P.G., Salmi, J., Eds.; World Bank: Washington, DC, USA, 2011; p. 381. ISBN 978-0-8213-9485-4. [ Google Scholar ] [ CrossRef ]
• Powell, J.J.W.; Fernandez, F.; Crist, J.T.; Dusdal, J.; Zhang, L.; Baker, D.P. Introduction: The Worldwide Triumph of the Research University and Globalizing Science. Century Sci. Int. Perspect. Educ. Soc. 2017 , 33 , 1–36. [ Google Scholar ] [ CrossRef ]
• Colina-Ysea, F.; Pantigoso-Leython, N.; Abad-Lezama, I.; Calla-Vásquez, K.; Chávez-Campó, S.; Sanabria-Boudri, F.M.; Soto-Rivera, C. Implementation of Hybrid Education in Peruvian Public Universities: The Challenges. Educ. Sci. 2024 , 14 , 419. [ Google Scholar ] [ CrossRef ]
• Fernandez, F.; Baker, D.P. Science Production in the United States: An Unexpected Synergy between Mass Higher Education and the Super Research University. Century Sci. Int. Perspect. Educ. Soc. 2021 , 33 , 85–111. [ Google Scholar ] [ CrossRef ]
• Jamil, S. The Challenge of Establishing World-Class Universities: Directions in Development ; World Bank: Washington, DC, USA, 2009; p. 115. ISBN 082-1378-767, 978-0821-3787-62. [ Google Scholar ] [ CrossRef ]
• Mudzakkir, M.; Sukoco, B.; Suwignjo, P. World-class Universities: Past and Future. Int. J. Educ. Manag. 2022 , 36 , 277–295. [ Google Scholar ] [ CrossRef ]
• Tian, L. Rethinking the global orientation of world-class universities from a comparative functional perspective. Int. J. Educ. Dev. 2023 , 96 , 102700. [ Google Scholar ] [ CrossRef ]
• Clark, B.R. Creating Entrepreneurial Universities: Organizational Pathways of Transformation ; Emerald Group Publishing Limited: London, UK, 1998; p. 200. ISBN 978-0 0804-3342-4. [ Google Scholar ]
• Etzkowitz, H. The Entrepreneurial University: Vision and Metrics. Ind. High. Educ. 2016 , 3 , 83–97. [ Google Scholar ] [ CrossRef ]
• Kazin, F.A.; Kondratiev, A.V. Development of the concept of an entrepreneurial university in Russian universities: New assessment tools. Univ. Upr. Prakt. Anal. Univ. Manag. Pract. Anal. 2022 , 26 , 18–41. (In Russian) [ Google Scholar ] [ CrossRef ]
• Berestov, A.V.; Guseva, A.I.; Kalashnik, V.M.; Kaminsky, V.I.; Kireev, S.V.; Sadchikov, S.M. The “national research university” project is a driver of Russian higher education. Vyss. Obraz. V Ross. High. Educ. Russ. 2020 , 29 , 22–34. (In Russian) [ Google Scholar ] [ CrossRef ]
• Matveeva, N.; Ferligoj, A. Scientific Collaboration in Russian Universities before and after the Excellence Initiative Project “5–100”. Scientometrics 2020 , 124 , 2383–2407. [ Google Scholar ] [ CrossRef ]
• Matveeva, N.; Sterligov, I.; Yudkevich, M. The Effect of Russian University Excellence Initiative on Publications and Collaboration Patterns. J. Informetr. 2021 , 15 , 101110. [ Google Scholar ] [ CrossRef ]
• Semenov, V.P.; Mikhailov Yu, I. Challenges and trends of quality management in the context of industrial and mineral resources economy. J. Min. Inst. 2017 , 226 , 497. [ Google Scholar ] [ CrossRef ]
• Litvinenko, V.S.; Bowbrick, I.; Naumov, I.A.; Zaitseva, Z. Global guidelines and requirements for professional competencies of natural resource extraction engineers: Implications for ESG principles and sustainable development goals. J. Clean. Prod. 2022 , 338 , 130530. [ Google Scholar ] [ CrossRef ]
• Semenova, T.; Martínez Santoyo, J.Y. Economic Strategy for Developing the Oil Industry in Mexico by Incorporating Environmental Factors. Sustainability 2024 , 16 , 36. [ Google Scholar ] [ CrossRef ]
• Rozhdestvensky, I.V.; Filimonov, A.V.; Khvorostyanaya, A.S. Methodology for assessing the readiness of higher educational institutions and scientific organizations for technology transfer. Innov. Innov. 2020 , 9 , 11–16. (In Russian) [ Google Scholar ] [ CrossRef ]
• Bakthavatchaalam, V.; Miles, M.; de Lourdes, M.-T.; Jose, S. Research productivity and academic dishonesty in a changing higher education landscape. On the example of technical universities in India (translated from English). Educ. Issues Educ. Stud. Mosc. 2021 , 2 , 126–151. (In Russian) [ Google Scholar ] [ CrossRef ]
• Aven, T. Perspectives on the nexus between good risk communication and high scientific risk analysis quality. Reliab. Eng. Syst. Saf. 2018 , 178 , 290–296. [ Google Scholar ] [ CrossRef ]
• Marinina, O.A.; Kirsanova, N.Y.; Nevskaya, M.A. Circular economy models in industry: Developing a conceptual framework. Energies 2022 , 15 , 9376–9386. [ Google Scholar ] [ CrossRef ]
• Siegel, D.; Bogers, M.; Jennings, D.; Xue, L. Technology transfer from national/federal labs and public research institutes: Managerial and policy implications. Res. Policy 2022 , 52 , 104646. [ Google Scholar ] [ CrossRef ]
• Snegirev, S.D.; Savelyev, V.Y. Quality management for scientific activities. Stand. I Kachestvo Stand. Qual. 2014 , 3 , 54–57. (In Russian) [ Google Scholar ]
• Leontyuk, S.M.; Vinogradova, A.A.; Silivanov, M.O. Fundamentals of ISO 9001:2015. J. Phys. Conf. Ser. 2019 , 1384 , 012068. [ Google Scholar ] [ CrossRef ]
• Wieczorek, O.; Muench, R. Academic capitalism and market thinking in higher education. In International Encyclopedia of Education , 4th ed.; Elsevier: Amsterdam, The Netherlands, 2023; pp. 37–47. [ Google Scholar ] [ CrossRef ]
• Overchenko, M.N.; Marinin, M.A.; Mozer, S.P. Quality improvement of mining specialists training on the basis of cooperation between Saint-Petersburg mining university and Orica company. J. Min. Inst. 2017 , 228 , 681. [ Google Scholar ] [ CrossRef ]
• Hernandez-Diaz, P.M.; Polanco, J.-A.; Escobar-Sierra, M. Building a measurement system of higher education performance: Evidence from a Latin-American country. Int. J. Qual. Reliab. Manag. 2021 , 38 , 1278–1300. [ Google Scholar ] [ CrossRef ]
• Zharova, A.; Karl, W.; Lessmann, H. Data-driven support for policy and decision-making in university research management: A case study from Germany. Eur. J. Oper. Res. 2023 , 308 , 353–368. [ Google Scholar ] [ CrossRef ]
• Lubango, L.M.; Pouris, A. Is patenting activity impeding the academic performance of South African University researchers? Technol. Soc. 2009 , 31 , 315–324. [ Google Scholar ] [ CrossRef ]
• de Jesus, C.S.; Cardoso, D.d.O.; de Souza, C.G. Motivational factors for patenting: A study of the Brazilian researchers profile. World Pat. Inf. 2023 , 75 , 102241. [ Google Scholar ] [ CrossRef ]
• Kiseleva, M.A. Development of an organizational and economic mechanism for managing the research activities of national research universities. Vestn. YURGTU NPI Bull. SRSTU NPI 2021 , 3 , 182–190. (In Russian) [ Google Scholar ] [ CrossRef ]
• Alakaleek, W.; Harb, Y.; Harb, A.A.; Shishany, A. The impact of entrepreneurship education: A study of entrepreneurial outcomes. Int. J. Manag. Educ. 2023 , 21 , 100800. [ Google Scholar ] [ CrossRef ]
• Rudko, V.A.; Gabdulkhakov, R.R.; Pyagay, I.N. Scientific and technical substantiation of the possibility for the organization of needle coke production in Russia. J. Min. Inst. 2023 , 263 , 795–809. Available online: https://pmi.spmi.ru/index.php/pmi/article/view/16246?setLocale=en_US (accessed on 10 September 2024).
• Gromyka, D.S.; Gogolinskii, K.V. Introduction of evaluation procedure of excavator bucket teeth into maintenance and repair: Prompts. MIAB. Mining Inf. Anal. Bull. 2023 , 8 , 94–111. (In Russian) [ Google Scholar ] [ CrossRef ]
• Michaud, J.; Turri, J. Values and credibility in science communication. Logos Epistem. 2018 , 9 , 199–214. [ Google Scholar ] [ CrossRef ]
• Oblova, I.S.; Gerasimova, I.G.; Goman, I.V. The scientific career through a gender lens: A contrastive analysis of the EU and Russia. Glob. J. Eng. Educ. 2022 , 24 , 21–27. [ Google Scholar ]
• Chiware, E.R.T.; Becker, D.A. Research trends and collaborations by applied science researchers in South African universities of technology: 2007–2017. J. Acad. Librariansh. 2018 , 44 , 468–476. [ Google Scholar ] [ CrossRef ]
• Palavesm, K.; Joorel, S. IRINS: Implementing a Research Information Management System in Indian Higher Education Institutions. Procedia Comput. Sci. 2022 , 211 , 238–245. [ Google Scholar ] [ CrossRef ]
• Litvinenko, V.S.; Petrov, E.I.; Vasilevskaya, D.V.; Yakovenko, A.V.; Naumov, I.A.; Ratnikov, M.A. Assessment of the role of the state in the management of mineral resources. J. Min. Inst. 2023 , 259 , 95–111. [ Google Scholar ] [ CrossRef ]
• Carillo, M.R.; Papagni, E.; Sapio, A. Do collaborations enhance the high-quality output of scientific institutions? Evidence from the Italian Research Assessment Exercise. J. Socio-Econ. 2013 , 47 , 25–36. [ Google Scholar ] [ CrossRef ]
• Chen, S.; Ren, S.; Cao, X. A comparison study of educational scientific collaboration in China and the USA. Phys. A Stat. Mech. Its Appl. 2021 , 585 , 126330. [ Google Scholar ] [ CrossRef ]
• Arpin, I.; Likhacheva, K.; Bretagnolle, V. Organising inter- and transdisciplinary research in practice. The case of the meta-organisation French LTSER platforms. Environ. Sci. Policy 2023 , 144 , 43–52. [ Google Scholar ] [ CrossRef ]
• Liew, M.S.; Tengku Shahdan, T.N.; Lim, E.S. Enablers in Enhancing the Relevancy of University-industry Collaboration. Procedia—Soc. Behav. Sci. 2013 , 93 , 1889–1896. [ Google Scholar ] [ CrossRef ]
• Tunca, F.; Kanat, Ö.N. Harmonization and Simplification Roles of Technology Transfer Offices for Effective University—Industry Collaboration Models. Procedia Comput. Sci. 2019 , 158 , 361–365. [ Google Scholar ] [ CrossRef ]
• Sciabolazza, V.L.; Vacca, R.; McCarty, C. Connecting the dots: Implementing and evaluating a network intervention to foster scientific collaboration and productivity. Soc. Netw. 2020 , 61 , 181–195. [ Google Scholar ] [ CrossRef ]
• Ovchinnikova, E.N.; Krotova, S.Y. Training mining engineers in the context of sustainable development: A moral and ethical aspect. Eur. J. Contemp. Educ. 2022 , 11 , 1192–1200. [ Google Scholar ] [ CrossRef ]
• Duryagin, V.; Nguyen Van, T.; Onegov, N.; Shamsutdinova, G. Investigation of the Selectivity of the Water Shutoff Technology. Energies 2023 , 16 , 366. [ Google Scholar ] [ CrossRef ]
• Mohamed, M.; Altinay, F.; Altinay, Z.; Dagli, G.; Altinay, M.; Soykurt, M. Validation of Instruments for the Improvement of Interprofessional Education through Educational Management: An Internet of Things (IoT)-Based Machine Learning Approach. Sustainability 2023 , 15 , 16577. [ Google Scholar ] [ CrossRef ]
• Tuan, N.A.; Hue, T.T.; Lien, L.T.; Van, L.H.; Nhung, H.T.T.; Dat, L.Q. Management factors influencing lecturers’ research productivity in Vietnam National University, Hanoi, Vietnam: A structural equation modeling analysis. Heliyon 2022 , 8 , e10510. [ Google Scholar ] [ CrossRef ]
• Akcay, B.; Benek, İ. Problem-Based Learning in Türkiye: A Systematic Literature Review of Research in Science Education. Educ. Sci. 2024 , 14 , 330. [ Google Scholar ] [ CrossRef ]
• Cherepovitsyn, A.E.; Tretyakov, N.A. Development of New System for Assessing the Applicability of Digital Projects in the Oil and Gas Sector. J. Min. Inst. 2023 , 262 , 628–642. Available online: https://pmi.spmi.ru/pmi/article/view/15795?setLocale=en_US (accessed on 10 September 2024).
• Murzo, Y.; Sveshnikova, S.; Chuvileva, N. Method of text content development in creation of professionally oriented online courses for oil and gas specialists. Int. J. Emerg. Technol. Learn. 2019 , 14 , 143–152. [ Google Scholar ] [ CrossRef ]
• Sveshnikova, S.A.; Skornyakova, E.R.; Troitskaya, M.A.; Rogova, I.S. Development of engineering students’ motivation and independent learning skills. Eur. J. Contemp. Educ. 2022 , 11 , 555–569. [ Google Scholar ] [ CrossRef ]
• Rijcke, S.D.; Wouters, P.F.; Rushforth, A.D.; Franssen, T.P.; Hammarfelt, B. Evaluation Practices and Effects of Indicator Use—A Literature Review. Res. Eval. 2016 , 25 , rvv038. [ Google Scholar ] [ CrossRef ]
• Cappelletti-Montano, B.; Columbu, S.; Montaldo, S.; Musio, M. New perspectives in bibliometric indicators: Moving from citations to citing authors. J. Informetr. 2021 , 15 , 101164. [ Google Scholar ] [ CrossRef ]
• García-Villar, C.; García-Santos, J.M. Bibliometric indicators to evaluate scientific activity. Radiología Engl. Ed. 2021 , 63 , 228–235. [ Google Scholar ] [ CrossRef ]
• Guskov, A.E.; Kosyakov, D.V. National factional account and assessment of scientific performance of organizations. Nauchnyye Tekhnicheskiye Bibl. Sci. Tech. Libr. 2020 , 1 , 15–42. (In Russian) [ Google Scholar ] [ CrossRef ]
• Khuram, S.; Rehman, C.A.; Nasir, N.; Elahi, N.S. A bibliometric analysis of quality assurance in higher education institutions: Implications for assessing university’s societal impact. Eval. Program Plan. 2023 , 99 , 102319. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Buehling, K. Changing research topic trends as an effect of publication rankings—The case of German economists and the Handelsblatt Ranking. J. Informetr. 2021 , 15 , 101199. [ Google Scholar ] [ CrossRef ]
• Kremcheev, E.A.; Kremcheeva, D.A. The content of the operation quality concept of the scientific and technical organization. Opcion 2019 , 35 , 3052–3066. [ Google Scholar ]
• Nyondo, D.W.; Langa, P.W. The development of research universities in Africa: Divergent views on relevance and experience. Issues Educ. Educ. Stud. Mosc. 2021 , 1 , 237–256. (In Russian) [ Google Scholar ] [ CrossRef ]
• Marozau, R.; Guerrero, M. Impacts of Universities in Different Stages of Economic Development. J. Knowl. Econ. 2016 , 12 , 1–21. [ Google Scholar ] [ CrossRef ]
• Fayomi, O.O.; Okokpujie, I.P.; Fayomi, O.S.I.; Udoye, N.E. An Overview of a Prolific University from Sustainable and Policy Perspective. Procedia Manuf. 2019 , 35 , 343–348. [ Google Scholar ] [ CrossRef ]
• Do, T.H.; Krott, M.; Böcher, M. Multiple traps of scientific knowledge transfer: Comparative case studies based on the RIU model from Vietnam, Germany, Indonesia, Japan, and Sweden. For. Policy Econ. 2020 , 114 , 102134. [ Google Scholar ] [ CrossRef ]
• See, K.F.; Ma, Z.; Tian, Y. Examining the efficiency of regional university technology transfer in China: A mixed-integer generalized data envelopment analysis framework. Technol. Forecast. Soc. Chang. 2023 , 197 , 122802. [ Google Scholar ] [ CrossRef ]
• Dusdal, J.; Zapp, M.; Marques, M.; Powell, J.J.W. Higher Education Organizations as Strategic Actors in Networks: Institutional and Relational Perspectives Meet Social Network Analysis. In Theory and Method in Higher Education Research ; Huisman, J., Tight, M., Eds.; Emerald Publishing Limited: Bingley, UK, 2021; Volume 7, pp. 55–73. [ Google Scholar ] [ CrossRef ]
• Silva, M.D.C.; de Mello, J.C.C.B.S.; Gomes, C.F.S.; Carlos, I.C. Efficiency analysis of scientific laboratories. Meta Aval. 2020 , 2 , 625–645. [ Google Scholar ] [ CrossRef ]
• Vinogradova, A.; Gogolinskii, K.; Umanskii, A.; Alekhnovich, V.; Tarasova, A.; Melnikova, A. Method of the Mechanical Properties Evaluation of Polyethylene Gas Pipelines with Portable Hardness Testers. Inventions 2022 , 7 , 125. [ Google Scholar ] [ CrossRef ]
• Chen, W.; Yan, Y. New components and combinations: The perspective of the internal collaboration networks of scientific teams. J. Informetr. 2023 , 17 , 101407. [ Google Scholar ] [ CrossRef ]
• Corcoran, A.W.; Hohwy, J.; Friston, K.J. Accelerating scientific progress through Bayesian adversarial collaboration. Neuron 2023 , 111 , 3505–3516. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wu, L.; Yi, F.; Huang, Y. Toward scientific collaboration: A cost-benefit perspective. Res. Policy 2024 , 53 , 104943. [ Google Scholar ] [ CrossRef ]
• Ilyushin, Y.; Afanaseva, O. Spatial Distributed Control System Of Temperature Field: Synthesis And Modeling. ARPN J. Eng. Appl. Sci. 2021 , 16 , 1491–1506. [ Google Scholar ]
• Cossani, G.; Codoceo, L.; Caceres, H.; Tabilo, J. Technical efficiency in Chile’s higher education system: A comparison of rankings and accreditation. Eval. Program Plan. 2022 , 92 , 102058. [ Google Scholar ] [ CrossRef ]
• Marinin, M.A.; Marinina, O.A.; Rakhmanov, R.A. Methodological approach to assessing influence of blasted rock fragmentation on mining costs. Gorn. Zhurnal 2023 , 9 , 28–34. [ Google Scholar ] [ CrossRef ]
• Sutton, E. The increasing significance of impact within the Research Excellence Framework (REF). Radiography 2020 , 26 (Suppl. S2), S17–S19. [ Google Scholar ] [ CrossRef ]
• Basso, A.; di Tollo, G. Prediction of UK Research excellence framework assessment by the departmental h-index. Eur. J. Oper. Res. 2022 , 296 , 1036–1049. [ Google Scholar ] [ CrossRef ]
• Groen-Xu, M.; Bös, G.; Teixeira, P.A.; Voigt, T.; Knapp, B. Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework. Res. Policy 2023 , 52 , 104729. [ Google Scholar ] [ CrossRef ]
• Reddy, K.S.; Xie, E.; Tang, Q. Higher education, high-impact research, and world university rankings: A case of India and comparison with China. Pac. Sci. Rev. B Humanit. Soc. Sci. 2016 , 2 , 1–21. [ Google Scholar ] [ CrossRef ]
• Shima, K. Changing Science Production in Japan: The Expansion of Competitive Funds, Reduction of Block Grants, and Unsung Heroe. Century Sci. Int. Perspect. Educ. Soc. 2017 , 33 , 113–140. [ Google Scholar ] [ CrossRef ]
• Li, D. There is more than what meets the eye: University preparation for the socio-economic impact requirement in research assessment exercise 2020 in Hong Kong. Asian Educ. Dev. Stud. 2021 , 11 , 702–713. [ Google Scholar ] [ CrossRef ]
• Radushinsky, D.A.; Kremcheeva, D.A.; Smirnova, E.E. Problems of service quality management in the field of higher education of the economy of the Russian Federation and directions for their solution. Relacoes Int. No Mundo Atual 2023 , 6 , 33–54. [ Google Scholar ]
• Shi, Y.; Wang, D.; Zhang, Z. Categorical Evaluation of Scientific Research Efficiency in Chinese Universities: Basic and Applied Research. Sustainability 2022 , 14 , 4402. [ Google Scholar ] [ CrossRef ]
• Cheng, Z.; Xiao, T.; Chen, C.; Xiong, X. Evaluation of Scientific Research in Universities Based on the Idea of Education for Sustainable Development. Sustainability 2022 , 14 , 2474. [ Google Scholar ] [ CrossRef ]
• Hou, L.; Luo, J.; Pan, X. Research Topic Specialization of Universities in Information Science and Library Science and Its Impact on Inter-University Collaboration. Sustainability 2022 , 14 , 9000. [ Google Scholar ] [ CrossRef ]
• Elbawab, R. University Rankings and Goals: A Cluster Analysis. Economies 2022 , 10 , 209. [ Google Scholar ] [ CrossRef ]
• Kifor, C.V.; Olteanu, A.; Zerbes, M. Key Performance Indicators for Smart Energy Systems in Sustainable Universities. Energies 2023 , 16 , 1246. [ Google Scholar ] [ CrossRef ]
• Guironnet, J.P.; Peypoch, N. The geographical efficiency of education and research: The ranking of U.S. universities. Socio-Econ. Plan. Sci. 2018 , 62 , 44–55. [ Google Scholar ] [ CrossRef ]
• Ma, Z.; See, K.F.; Yu, M.M.; Zhao, C. Research efficiency analysis of China’s university faculty members: A modified meta-frontier DEA approach. Socio-Econ. Plan. Sci. 2021 , 76 , 100944. [ Google Scholar ] [ CrossRef ]
• Tavares, R.S.; Angulo-Meza, L.; Sant’Anna, A.P. A proposed multistage evaluation approach for Higher Education Institutions based on network Data envelopment analysis: A Brazilian experience. Eval. Program Plan. 2021 , 89 , 101984. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Le, M.H.; Afsharian, M.; Ahn, H. Inverse Frontier-based Benchmarking for Investigating the Efficiency and Achieving the Targets in the Vietnamese Education System. Omega 2021 , 103 , 102427. [ Google Scholar ] [ CrossRef ]
• Adot, E.; Akhmedova, A.; Alvelos, H.; Barbosa-Pereira, S.; Berbegal-Mirabent, J.; Cardoso, S.; Domingues, P.; Franceschini, F.; Gil-Doménech, D.; Machado, R.; et al. SMART-QUAL: A dashboard for quality measurement in higher education institutions. Qual. Reliab. Manag. 2023 , 40 , 1518–1539. [ Google Scholar ] [ CrossRef ]

Share and Cite

Radushinsky, D.A.; Zamyatin, E.O.; Radushinskaya, A.I.; Sytko, I.I.; Smirnova, E.E. The Performance and Qualitative Evaluation of Scientific Work at Research Universities: A Focus on the Types of University and Research. Sustainability 2024 , 16 , 8180. https://doi.org/10.3390/su16188180

Radushinsky DA, Zamyatin EO, Radushinskaya AI, Sytko II, Smirnova EE. The Performance and Qualitative Evaluation of Scientific Work at Research Universities: A Focus on the Types of University and Research. Sustainability . 2024; 16(18):8180. https://doi.org/10.3390/su16188180

Radushinsky, Dmitry A., Egor O. Zamyatin, Alexandra I. Radushinskaya, Ivan I. Sytko, and Ekaterina E. Smirnova. 2024. "The Performance and Qualitative Evaluation of Scientific Work at Research Universities: A Focus on the Types of University and Research" Sustainability 16, no. 18: 8180. https://doi.org/10.3390/su16188180

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

An official website of the United States government, Department of Justice.

Here's how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Research, Evaluation, and Analysis of Call Handling on Three-Digit Hotlines (REACH-3D)

Award information.

CHICAGO , IL

Description of original award (Fiscal Year 2024, $935,000)

America’s 911 and alternative three-digit service-hotline universe comprises a patchwork of about 10,000 specific systems that face significant funding, staffing, coordination, and operational challenges. This field is thus fragmented and poorly understood. University of Chicago Health Lab, partnering with National Association of State 911 Administrators (NASNA), Inform USA (formerly Alliance of Information and Referral Systems or AIRS), United Way, Medical College of Wisconsin, the Emergency Medical Services (EMS) Eagles Global Alliance (The Eagles), International Association of Firefighters (IAFF), National Alliance on Mental Illness (NAMI), Native and Strong Lifeline, TDIforAccess (TDI, formerly Telecommunications for the Deaf and Hard of Hearing), The Arc For People with Intellectual and Development Disabilities (The Arc), NORC (National Opinion Research Center), and others, seek to address these gaps. 

Research, Evaluation, and Analysis of Call Handling on Three-Digit Hotlines (REACH-3D) includes meaningful engagement with those closest to the problem, drawing on the perspectives of practitioners and community members. REACH-3D’s multi-disciplinary team includes researchers, practitioners, and community members with lived experience and expertise spanning emergency crisis response, deaf and hard of hearing accessibility, North American Indigenous populations and the challenges they face, intellectual and developmental disabilities, public health, public policy, medicine, criminology, social work, policing, public opinion research, and relevant technologies. REACH-3D will assess these hotlines’ discrete and mutual ability to safely, equitably, efficiently, and effectively meet public health and safety needs, while avoiding unnecessary or unwarranted criminal legal system involvement. 

REACH-3D employs a survey and case studies approach to gathering the most extensive and comprehensive data ever collected regarding the 911 landscape, alternative three-digit service lines, and their accompanying interactions to better-meet public needs for government and community-based services. Project activities include extensive literature reviews; an assembly and analysis of three-digit lines’ operational, training, and programmatic data; surveys of call-line staff and leadership; surveys of policymakers and the general public; and case studies of communities in particular need of accompanying services. Together, these activities will provide a clearer understanding of the practical issues and opportunities facing three-digit service lines. 

REACH-3D will immediately enhance the understanding of these disparate systems and identify the comprehensive challenges these systems confront, including those related to funding, staffing, operations, and coordination. REACH-3D’s findings will provide scientific opportunities for high-quality peer-review publications. These will also provide explicit guidance for local, state, and regional practitioners, policymakers, and researchers, which will be developed and disseminated via practitioner and policymaker guidance, briefs, toolkits, and popular press. CA/NCF

Similar Awards

• Leisure Risk for Youth on Probation: How it Relates to Recidivism and How Probation Officers Address it in Case Planning
• The Ecology of Resilience: Examining Impacts of Service Engagement, Facility Safety, and Trauma History on Positive Life Trajectories in Justice-Involved Youth
• Collaborative Strategies in Safeguarding Children: A Community-Centric Approach to Overdose Response

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 14, Issue 9
• Construction of a competency evaluation model for Clinical Research Coordinators in China: a study based on the Delphi method and questionnaire survey
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Yanhong Zhu 1 ,
• Xiang Sun 2 ,
• Maohang Cai 1 ,
• Jun Shi 1 ,
• http://orcid.org/0000-0003-3951-8061 Xin Li 1 , 3 , 4
• 1 School of Health Policy and Management , Nanjing Medical University , Nanjing , Jiangsu Province , China
• 2 Department of Expanded Program on Immunization , Jiangsu Provincial Center for Disease Control and Prevention , Nanjing , Jiangsu Province , People's Republic of China
• 3 Department of Clinical Pharmacy , Nanjing Medical University , Nanjing , Jiangsu Province , China
• 4 Center for Global Health, School of Public Health , Nanjing Medical University , Nanjing , Jiangsu Province , China
• Correspondence to Dr Xin Li; xinli{at}njmu.edu.cn

Purpose As the number of clinical trials in China continues to grow, the assessment of competency of Clinical Research Coordinators (CRCs), who play a crucial role in clinical trials, has become an important and challenging topic. This study aims to construct a competency model for CRCs tailored to the Chinese context, in order to promote the standardisation and regulated development of the CRC industry.

Study design and setting This study was conducted in China, engaging CRCs as the primary subjects. A competency evaluation model for CRCs was constructed through literature review, semi-structured interviews, Delphi expert consultation and the analytic hierarchy process. A questionnaire survey was distributed to a broad sample of CRCs across China to evaluate the model’s reliability and validity.

Results The final model encompasses 4 core competency dimensions and 37 indicators, tailored to assess the competencies of CRCs in China. The questionnaire yielded an effective response rate of 81.83%, with high internal consistency(Cronbach’s α>0.7). Factor analysis confirmed the model’s structure, indicating good reliability and validity.

Conclusion This study represents a pioneering effort in constructing a competency model specifically designed for Chinese CRCs, complemented by a robust and valid assessment scale. The findings bear significant implications for the recruitment, training, development and management of CRCs.

• clinical competence
• clinical decision-making
• clinical trial
• education & training (see medical education & training)
• education, medical

Data availability statement

No data are available.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2024-083957

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

The study uses a robust Delphi method and questionnaire survey to construct a competency model, enhancing the model’s relevance and authority in the Chinese context.

Advanced statistical analyses, including factor analysis and the analytic hierarchy process, are applied to ensure the model’s scientific rigour and reliability.

The research provides a pioneering framework specifically tailored for Chinese Clinical Research Coordinators (CRCs), which is a significant contribution to the professional development and standardisation of the CRC industry in China.

The use of convenience sampling might affect the generalisability of the findings, potentially limiting the representation of China’s full regional diversity.

While the model has been validated through statistical analysis, further empirical testing across various clinical research settings in China is needed to confirm its broader applicability.

Introduction

Clinical trials are an essential phase in drug development, with the efficacy and safety of new drugs ultimately confirmed through human clinical trials. 1 As key roles in clinical trials, Clinical Research Coordinators (CRCs) 2 3 assist investigators with non-medical judgement related tasks, such as trial management and daily coordination activities, acting as a bridge between clinical researchers, sponsors and participants. 4–7 In China, investigators face particularly heavy daily clinical work, which often makes it difficult to devote enough time to the entire clinical trial. CRCs, as assistants to investigators, have become an indispensable role in clinical trials. With the rapid growth in the number of clinical trials in China, the role of CRCs has become increasingly significant.

However, unlike their counterparts in the USA and Europe, where CRCs undergo formal training and certification processes, CRCs face challenges due to the lack of a standardised training and certification system in China.

In Europe and the USA, CRCs, as a profession, need to undergo formal training courses provided by recognised professional institutions or universities before entering the industry. After graduation, they can participate in clinical trials. Two years into their careers, they are required to take professional qualification examinations organised by organisation like the Society for Clinical Research Associates 8 or the Association of Clinical Research Professionals. 9 10 Currently, these two organisations conduct CRC certification examinations in 15 countries and regions in Europe, the Americas and Asia, with a mandatory requirement of at least 2 years of CRC work experience for examination candidates. Throughout their careers, CRCs must continuously attend workshops and training sessions organised by professional institutions or universities, covering clinical trial skills, medical ethics, pharmacy and specialised knowledge, among other subjects. Regular continuing education is necessary to enhance job capabilities. In contrast, there is currently no unified management standard or model for CRCs in China. Both regulatory oversight and industry norms lack clear and unified requirements and entry standards. Moreover, as CRCs typically come from different Site Management Organisations, the diversity in their origins leads to a disparity in personnel quality. The industry lacks specialised training, and assessment system at the national level. The professional recognition of CRC practitioners is insufficient, and their career development paths are not clearly defined, leading to a high turnover rate. This poses a challenge to the management and efficiency of clinical trial institutions. During the evaluation process of clinical trials, there is seldom consideration given to the competencies of CRCs and suggestions to improve their competencies. This leads to a lack of targeted training and guidance, which further exacerbates the turnover of CRCs and has a detrimental impact on the overall quality of clinical trials. 11

China’s unique cultural and regulatory environment requires a tailored model for assessing the competencies of CRCs, which existing frameworks like The Joint Task Force for Clinical Trial Competency model 12 13 may not adequately address with the required specificity for the Chinese healthcare context. Specifically, the vast geographical spread of the country leads to significant disparities in education and healthcare systems between urban and rural areas, which in turn influence the accessibility and delivery of clinical trials. Urban centres typically offer more advanced medical facilities and a higher concentration of healthcare professionals, while rural areas might face limitations in infrastructure and resources. Moreover, the Chinese healthcare system is characterised by its own set of regulations, policies and cultural norms that shape the conduct and management of clinical research. For instance, the hierarchical structure of medical institutions, the operation of the healthcare insurance system, and the specific regulatory pathways for drug and device approvals are all context-specific factors that CRCs must navigate. Furthermore, the cultural values and social dynamics in China can significantly affect patient recruitment, informed consent processes and the ethical considerations of clinical trials. These cultural factors, along with the unique educational background and professional development needs of CRCs in China, highlight the importance of a competency model that is not only context-specific but also sensitive to regional variations and local practices.

Therefore, the existing JTF model, while valuable in its general approach to clinical research competencies, may not fully capture the nuances of working within China’s healthcare system, including the rural-urban divide and the specific regulatory and cultural demands. The need for a competency model for CRCs tailored to the Chinese context is evident. This study aims to construct a competency model for CRCs that is not only reflective of the current clinical research environment in China but also serves as a benchmark for the professional development, training and evaluation of CRCs in the country.

Design and procedures

Our study was conducted from January to December 2023, and the construction of the CRC general competency model combined the advantages of qualitative and quantitative analysis, divided into three steps. Online supplemental figure illustrates the technical workflow of our study.

Supplemental material

Competency characteristic system.

Initially, we conducted a comprehensive literature review using keywords such as “Clinical Research Coordinator”, “Study Nurse”, “competency model” and “Delphi method”, in databases including PubMed, Web of Science, the China Biomedical Literature Database and CNKI (China National Knowledge Infrastructure). The review focused on existing competency frameworks and studies related to CRCs, particularly within the Chinese context.

The process of identifying the key competency indicators for CRCs in China was grounded in Competency-Based Theories, which emphasise characteristics significantly correlated with job performance, such as knowledge, skills, abilities, traits and motivations. 14 To systematically categorise these characteristics, we applied the Onion Model Theory. This model metaphorically describes competencies in layers, from the outermost layer representing basic knowledge and skills, through the middle layer encompassing attitudes and values along with social roles and self-perception, to the innermost layer reflecting personality and motivation—the deeper, core competencies that are more challenging to cultivate, assess and modify. 15 In alignment with the Onion Model, we preliminarily categorised the CRC competencies into five primary indicators, corresponding to the different layers of the model: Knowledge, Skills, Attitudes/Values, Personality and Motivation.

To develop a deeper understanding of the roles and competencies of CRCs in China, we conducted semi-structured interviews with 15 CRCs who have more than 5 years of clinical research experience. The interviews aimed to explore the actual clinical competencies and challenges faced by CRCs in their daily work. The interviews lasted for 60–120 min, continuing until information saturation was reached, and no new information emerged. Core questions included, ‘In the process of clinical trials, what qualities and abilities do you believe CRCs should possess, and which ones are the most important?’; ‘In clinical trials or the assessment of CRC work, what factors are typically considered to evaluate whether a CRC is outstanding?’; ‘When conducting interviews during projects or personnel recruitment for CRCs, what points are taken into consideration, and which do you believe are essential and closely related to hiring decisions?’. The semi-structured interviews were digitally recorded and transcribed verbatim. We used thematic analysis approach to analyse the interview data, organised and summarised the competency characteristics of CRCs, and integrated them with the characteristics previously identified through literature review.

Construction of initial CRC competency model

Subsequently, based on the aforementioned competency characteristic system, the Delphi method was employed to reach a consensus among experts regarding the competency model for CRCs. The typical number of experts in a Delphi survey usually falls between 10 and 50 individuals. 16 17 For this study, the Delphi survey expert panel was specifically constituted from the clinically advanced eastern region of China, comprising a select group of 16 experts ( online supplemental table 1 ). Inclusion criteria comprised: having more than 5 years of experience in drug clinical trials, possessing rich operational experience in drug clinical trials, being familiar with the job responsibilities and work content of CRCs, and having a high level of interest and enthusiasm for the content of this consultation, as well as the ability to continuously and completely complete the survey and consultation of this study. The Delphi questionnaire was designed to assess the importance, feasibility and sensitivity of each competency indicator using a 5-point Likert scale. 18 After the first round of consultations, we analysed the results based on the mean scores and the coefficient of variation (CV) for each indicator across the three dimensions. Indicators were retained, modified or deleted according to the established criteria: if two or more dimensions had a mean score of ≤3.5 or a CV ≥25%, the indicator was considered for deletion. If only one dimension met these criteria, it was discussed further with the experts for a decision on modification or retention. An indicator was included if it had a mean score >3.5 and a CV <25% across all three dimensions. 19 The second round of the Delphi process incorporated the feedback from the first round, and the questionnaire was revised accordingly. The same experts were invited to provide their input again, ensuring continuity and a thorough examination of the competency model. The process continued until a consensus was reached among the experts, indicating a stable and agreed-upon set of competency indicators for CRCs in China.

Establishment of the final CRC competency model

Third, to avoid sequence bias, we randomly arranged the items in the initial competency model established by the Delphi panel, and assessed the importance of each item using a Likert scale. This study conducted a convenient sample of CRCs within clinical trial teams. Participants were sourced from CRCs working in drug clinical trial institutions approved by the National Medical Products Administration across 30 provinces in mainland China. According to the sample size calculator, the minimum estimated sample size Raosoft for this survey was 377, using the formula, where n is the required sample size, N is the population size, x is the CI, assuming a 95% CI and E is the margin of error at 5%: n=N×x/((N−1) E 2 +x). 20 Two trained research assistants from each province distributed the online questionnaire to potential participants. Participants completed the online survey through the ‘Survey Star’ online survey platform (Changsha Ranxing Technology, China). The questionnaire was completed anonymously, with no personal identification information. All participants were informed about the study before accessing the online questionnaire and provided consent before commencing. Participants were asked to rate items on a 5-point Likert scale, where 1–5 represented ‘not important at all’, ‘unimportant’, ‘neutral’, ‘important’ and ‘very important’, respectively. The questionnaire data underwent tests for reliability and validity to validate the scientific and reliable nature of the competency model, ultimately resulting in the final CRC competency model.

Patient and public involvement

This study did not involve patients or the public in the development of the research question, design, recruitment or conduct of the study. The research was focused on constructing a competency evaluation model for CRCs in China. Results will be disseminated through academic channels. There were no patient advisers involved in this study.

Statistical analysis

Data management and analysis were conducted using a combination of software tools to ensure robust and accurate results. Microsoft Excel was used for preliminary data organisation, while SPSS software V.18.0 was employed for advanced statistical analysis. In the Delphi study, Kendall’s coefficient was employed as a parameter to assess the consistency of opinions among different experts. To calculate weights and to reflect the relative importance of each indicator, we used the analytic hierarchy process (AHP) and the combination weighting method. These weights determined through AHP provide a reference value for the application of the competency model. Subsequently, we employed survey data to test the reliability and validity of the competency model. Cronbach’s α coefficient, commonly used to assess the internal consistency of a scale, was used to examine the internal consistency of both the entire questionnaire and the four primary indicators. An α coefficient above the standard threshold of 0.7 indicates good internal consistency and reliability. Simultaneously, we conducted exploratory factor analysis (EFA) to explore the underlying factor structure of the competency model. This was followed by confirmatory factor analysis (CFA) using AMOS software V.22.0, which allowed us to test the effectiveness of our model. The CFA results were evaluated using a set of goodness-of-fit indices, which are recommended for assessing the model’s adequacy. These indices included the χ 2 /df ratio(CMIN/DF), Goodness of Fit Index (GFI), Adjusted Goodness of Fit Index (AGFI), Normed Fit Index (NFI), Incremental Fit Index (IFI), Tucker-Lewis Index (TLI), Comparative Fit Index (CFI) and Root Mean Square Error of Approximation (RMSEA). A well-fitting model is generally indicated by values close to or exceeding the recommended thresholds: CMIN/DF <3, GFI, AGFI, NFI, IFI, TLI and CFI >0.9 and RMSEA <0.08. 21 22 A significance level of p<0.05 was deemed statistically significant.

Semistructured interviews

The interviews conducted indeed captured narratives that reflect the unique aspects of Chinese culture, healthcare and clinical trials management. These narratives provided insights into the challenges faced by CRCs. Through integration and optimisation, the preliminary results of the discussions led to the formation of 5 primary indicators, 22 secondary indicators and 61 tertiary indicators along with their explanations. The demographic data of the interviewees can be found in online supplemental table 2 .

Delphi panels

After two rounds of Delphi consultations, a consensus was reached among the panel members. The Kendall’s coefficient of concordance for the importance, feasibility and sensitivity of the indicators in the first round were 0.315, 0.244 and 0.202, respectively (p<0.001). In the second round, the coefficients were 0.297, 0.306 and 0.212 (p<0.001), confirming the consensus among experts. Based on the outcomes of the first round, 22 tertiary indicators were removed. Additionally, four tertiary indicators were merged into two (‘adjustment ability’ and ‘adaptability’ into ‘adjustment and adaptability’, ‘analytical ability’ and ‘evaluation ability’ into ‘evaluation and analysis ability’), and ‘research-related knowledge’ was split into ‘product knowledge’ and ‘protocol knowledge’. This process resulted in a refined set of 5 primary indicators, 21 secondary indicators and 38 tertiary indicators that were carried forward to the second round. In the second round, the primary indicator ‘Motivation’ was further removed due to its mean scores in feasibility and sensitivity being <3.5. Ultimately, the CRCs competency model was established with 4 primary indicators, 20 secondary indicators and 37 tertiary indicators, which are presented in the final model. Then we used the AHP and combination weighting to calculate the competency indicators’ weights. In the primary indicators, ‘Knowledge’ weight is the highest at 0.4715, followed by ‘Attitudes/Values’ with a weight coefficient of 0.2550, ‘Personality’ with a weight coefficient of 0.1653 and ‘Skills’ with the lowest weight, at 0.1083. The weight results for the secondary and tertiary indicators can be found in table 1 .

• View inline

Competency model hierarchical indicators and their weights

Survey participants and questions

From May to October 2023, a total of 600 questionnaires were distributed, resulting in 546 completed questionnaires. After excluding invalid questionnaires (due to incomplete data), a total of 491 valid questionnaires were obtained, with an effective response rate of 81.83%. The demographic characteristics of the survey participants are presented in table 2 .

General demographic characteristics of respondents

Questionnaire reliability and validity analysis

This study identified four core factors of CRC competencies: Knowledge, Skills, Attitudes/Values and Personality. A reliability analysis was conducted for each factor ( table 3 ). The Cronbach’s α coefficients exceeded the benchmark of 0.7 for all factors, indicating good internal consistency and reliability. The corrected item-total correlation for all items surpassed the threshold of 0.5, indicating that the measured items meet the research requirements. Further analysis showed that the deletion of any item would not increase the Cronbach’s α value, substantiating the reliability of the variables.

Reliability analysis

For the factor analysis, the data set was randomly split into two groups. The first group was subjected to EFA to examine the underlying factor structure of the competencies. With a Kaiser-Meyer-Olkin measure of 0.972 and a significant Bartlett’s Test of Sphericity (p<0.001), the suitability of the questionnaire for EFA was confirmed. Using the principal component extraction method and following a maximum orthogonal rotation, four distinct factors were identified, each with factor loadings exceeding 0.5 ( table 4 ). These factors accounted for 30.933%, 12.279%, 10.494% and 9.387% of the variance, respectively, culminating in a cumulative explained variance of 63.093%. The EFA results were statistically significant.

Rotated component matrix

The second data group underwent CFA to verify the structural model. Employing AMOS software for the CFA, the CFA model diagram is shown in figure 1 , and the goodness-of-fit indices are as follows: CMIN/DF=1.541, GFI=0.903, AGFI=0.891, NFI=0.925, IFI=0.972, TLI=0.970, CFI=0.972 and RMSEA=0.033. All these indices meet the model fit standards, demonstrating a good fit for the model.

• Download figure
• Open in new tab
• Download powerpoint

Results of confirmatory-factor analysis of general competency model for Clinical Research Coordinator.

The development of the CRC competency model in this study addresses a significant gap in the Chinese clinical research context. Given the rapid increase in clinical trials, the role of CRCs has become not only pivotal but also increasingly complex. This necessitates a structured and scientifically rigorous approach to defining and evaluating their competencies, which our model aims to provide.

The competency model developed in this study is underpinned by a rigorous scientific approach. It incorporates a comprehensive literature review and expert consultations, ensuring that the indicators are grounded in both theoretical frameworks and practical insights from the field of clinical research in China. Our model takes into account the unique aspects of China’s healthcare system and cultural nuances, which are crucial for the effective implementation of CRC competencies. It acknowledges the unique cultural and systemic factors that influence the role of CRCs, making it a more relevant tool for assessing competencies in this setting compared with generic or Western-centric models.

Our CRC competency model has been thoughtfully designed to take into account the unique aspects of Chinese culture, with targeted considerations in various respects. The ‘Attitudes/Values’ primary index within our model places special emphasis on ‘Integrity’, reflecting the high regard for ethical conduct and compliance within the context of clinical research in China. 23 Additionally, the indices of ‘Patient Education Skills’ and ‘Subject Focus’ embody a patient-centred approach that aligns with the expectations of healthcare in China. The ‘Professional Knowledge’ index includes a deep understanding of medical practices and policies in China, ensuring that CRCs can adapt to the local medical environment. Furthermore, the ‘Process Knowledge’ index covers site and subject management processes that are consistent with the regulatory requirements for clinical trial management in China. The ‘Language Proficiency’ index recognises the importance of effective communication across the diverse linguistic backgrounds present in China. The indices of ‘Adaptability and Adjustment’ and ‘Boundary Awareness’ highlight the need for CRCs to possess flexibility and a clear understanding of professional boundaries within the cultural and healthcare context of China. At the same time, the ‘Legal and Regulatory’ index ensures that CRCs have a profound understanding of China’s unique laws and regulations, which is crucial for protecting the rights and interests of subjects and the compliance of clinical trials. 24 Through these comprehensive considerations of indicators, our model not only respects Chinese cultural values and social expectations but also adapts to the actual situation of China’s medical system and regulatory environment.

In comparison with existing models, such as the internationally used JTF competency model, 25 our model distinctly emphasises familiarity with Chinese laws and regulations, a patient-centred approach, and a high valuation of ethical behaviour and compliance. In addition, considering the uneven distribution of public medical resources in China’s medical system, our model enhances the assessment of CRCs’ ability to work in medical institutions at different levels, ensuring that they can effectively coordinate clinical research in a diverse medical environment. We particularly emphasise ‘Patient Education Skills’ and ‘Subject Focus’ to reflect the respect for patients and patient-centred medical methods in Chinese culture. In addition, our model gives higher weight to ‘Integrity’ in the ‘Attitudes/Values’ aspect, reflecting the strict requirements of China’s clinical research on ethics and regulatory compliance.

Although our model does not explicitly use the term ‘research ethics’, it integrates the concept of research ethics and subject protection through specific third-level indicators. For instance, the ‘Research Related Knowledge’ indicator stresses the importance for CRCs to understand the trial’s purpose, target population, procedures and observation indicators, as well as the fundamental components and key points of informed consent, which are integral to research ethics. Our model includes considerations for ‘Centre Management Processes’, such as research institution establishment, ethical review processes, contract review and signing procedures and safety event reporting processes, all of which are pivotal for the protection of research ethics and subject rights. Furthermore, the ‘Subject Management Process’ is a pivotal component of our model, highlighting the significance of CRCs’ management and attention to subjects during the trial process. A robust subject management process is essential not only for safeguarding the ethics and regulatory compliance of clinical trials but also for enhancing subject participation and ensuring the accuracy of trial data. In addition to the subject management processes, our model places considerable emphasis on the indicators of ‘Legal and Regulatory’, ‘Legal Adherence’ and ‘Compliance’. These indicators reflect the critical role that CRCs’ knowledge of and adherence to laws and regulations play in ensuring the ethical conduct of clinical trials. A thorough understanding and strict compliance with legal frameworks are as vital as the management processes themselves, providing a dual foundation for the ethical and compliant execution of clinical research.

Strengths and limitations of this study

While the CRC competency scale developed in this study demonstrates strengths in construction methods, content coverage, reliability and validity, there are inherent limitations that should be acknowledged. Potential regional or institutional biases in the sample, incompleteness of model dimensions and scale items, and insufficient validation of the model highlight the need for cautious application and future refinement and validation in subsequent research. The reliance on a convenience sample may limit the generalisability of the findings. Future research should aim to diversify the sample and further validate the model across various clinical research settings in China.

This study marks the inaugural development of a competency model tailored for Chinese CRCs, accompanied by a reliable and valid assessment scale. The findings bear significant implications for the recruitment, training, development and management of CRCs. The results offer valuable guidance for CRCs in their professional development, aiding them in acquiring essential knowledge and skills. Moreover, they provide reference points for CRC managers, assisting in the selection and cultivation of CRC talent. In the long run, these advancements contribute to the professionalisation and standardisation of clinical research, ultimately enhancing the quality of healthcare services and patient health outcomes.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

Ethical approval for this study was provided by the Ethics Committee of Nanjing Medical University, China (Approval No. (2021)103). The experts were fully informed of the purpose, significance, research contents and methods of the study. The freedom and rights of the participants to participate in or withdraw from the study were respected, and their interests were protected. To protect the experts from any consequences, data were made anonymous before analyses.

Acknowledgments

The authors wish to thank all participants, especially the 16 experts who helped with the analysis and interpretation of this study.

• Poston RD ,
• Buescher CR
• Fujiwara N ,
• Shirai Y , et al
• Yanagawa H ,
• Nokihara H ,
• Yokoi H , et al
• Jeong JH , et al
• Wang J , et al
• Society Of Clinical Research Associates (SOCRA)
• ↵ Training, continuous education and career path of clinical research coordinator/research nurse . Chin J New Drugs Clin Rem 2018 ; 37 : 90 – 4 . OpenUrl
• Sonstein SA ,
• Namenek Brouwer RJ ,
• Gluck W , et al
• Chouhan VS ,
• Srivastava S
• Boyatzis RE
• Meretoja R ,
• Vyagusa DB ,
• Mubyazi GM ,
• Jin X , et al
• ↵ Raosoft sample size calculator . n.d. Available : http://www.raosoft.com/samplesize.html
• Jackson DL ,
• Gillaspy JA ,
• Purc-Stephenson R
• Chang Q , et al
• Johnson EA ,
• Ma J , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2
• Data supplement 3

YZ and XS are joint first authors.

Contributors XL conceived and designed the study. YZ, MHC and JS designed the questionnaire and collected data. XS performed the statistical analysis. YZ drafted the manuscript. XL responsible for the revision of the manuscript. All authors have read and agreed to the published version of the manuscript. XL is the guarantor.

Funding This study was supported by grants from the National Natural Science Foundation of China (No. 72074123), the Soft Science Project of Jiangsu Science and Technology Department (No. BR2023018-5).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• News & Events
• OTP Events, Meetings, and Workshops
• Workshop on Integration Site Analysis During Long Term Follow-Up for Gene Therapies with Integrating Viral Vectors - 11/14/2024

Workshop | Virtual

Event Title Workshop on Integration Site Analysis During Long Term Follow-Up for Gene Therapies with Integrating Viral Vectors November 14, 2024

The FDA’s Center for Biologics Evaluation and Research (CBER) Office of Therapeutic Products (OTP) is hosting a virtual scientific public workshop on the clinical use of integration site analysis (ISA) during long term follow-up following administration of gene therapies with integrating viral vectors. In this workshop, FDA will convene a panel of external experts to discuss the risk of insertional mutagenesis and best practices for ISA method design, data analysis, and clinical interpretation. The workshop participants will include research scientists with experience in vector design and ISA, as well as clinician-scientists with expertise in clinical interpretation of ISA data.

Meeting Logistics and Registration:

• Date: Thursday, November 14, 2024
• Time: 9:30 a.m. to 5:00 p.m. EST
• Location: The webinar will be held via Zoom.
• Registration: Registration is required. Please register for the event now.

Focus for this Workshop

As of 2024, twelve approved and hundreds of investigational cell-based gene therapies use integrating viral vectors. While stable integration may have considerable therapeutic benefits, integration into genomic DNA can alter host gene expression and may contribute to secondary malignancies via insertional mutagenesis. Additionally, the choice of vector (e.g., lentivirus versus gammaretrovirus) and cell substrate (e.g., T cells versus CD34+ cells) may also contribute to this risk.

ISA is the established approach to identify vector insertion sites. However, ISA method design and data interpretation are highly complex and product-specific. Moreover, ISA methods may be costly and challenging to perform. There is a critical need to understand the best practices for product design and ISA method design, execution, and interpretation to facilitate efficient evaluation of complex gene therapy products and protect the public from potential serious adverse events.

This workshop will include discussions on the following topics:

• Effects of Cell Substrate and Vector Design on the Risk of Vector Insertion-Related Adverse Events
• ISA Methodology, Assay Design, and Analytical Capabilities
• ISA Analysis, Quality Control, and Data Reporting
• Clinical Follow-Up for Integrating Viral Vectors
• Using ISA to Inform Clinical Decisions

For current guidance on the role of ISA in clinical follow-up, please refer to section V.F “Special Considerations Regarding Integrating Vectors” of the FDA Guidance for Industry “Long Term Follow-Up After Administration of Human Gene Therapy Products” dated January 2020 ( https://www.fda.gov/media/113768/download ).

Stay Connected

Sign up for the CBER listserv or follow us on social media to stay up to date on all FDA CBER news and events.

Redirect Notice

Inclusion of women and minorities as participants in research involving human subjects.

Learn about the policy for the Inclusion of Women and Minorities in NIH-funded research and how to comply with this policy in applications and progress reports.

NIH is mandated by the Public Health Service Act sec. 492B, 42 U.S.C. sec. 289a-2 to ensure the inclusion of women and members of racial and ethnic minority groups in all NIH-funded clinical research in a manner that is appropriate to the scientific question under study. The primary goal of this law is to ensure that research findings can be generalizable to the entire population. Additionally, the statute requires clinical trials to be designed to analyze whether study outcomes differ for women and members of racial and ethnic minority groups.

Implementation

Applications & proposals.

All NIH-funded studies that meet the NIH definition for clinical research must address plans for the inclusion of women and minorities within the application or proposal. Using the PHS Human Subjects and Clinical Trial Information Form, applications and proposals should describe the composition of the proposed study population in terms of sex or gender, racial, and ethnic groups, and provide a rationale for the proposed section. Any exclusions based on sex or gender, race, or ethnicity must include a rationale and justification based on a scientific or ethical basis. Investigators should also plan for appropriate outreach programs and activities to recruit and retain the proposed study population consistent with the purposes of the research project. Refer to the PHS Human Subjects and Clinical Trial Information Form Instructions for complete guidance on what to address in your application.

Peer Review

Scientific Review Groups will assess each application/proposal as being "acceptable" or "unacceptable" with regard to the inclusion of racial and ethnic minorities and women in the research project. For additional information on review considerations, refer to the Guidelines for the Review of Inclusion in Clinical Research . For information regarding the coding used to rate inclusion during peer review, see the list of NIH Peer Review Inclusion Codes .

Progress Reports

NIH recipients/offerors must collect and annually report information on sex or gender race, and ethnicity in progress reports. Refer to this Decision Tree for help determining reporting expectations for different types of studies.

Special Considerations for NIH-defined Phase III Clinical Trials

Applications & Proposals: If the proposed research includes an NIH-defined Phase III Clinical Trial , evidence must be reviewed to show whether or not clinically important differences in the intervention effect by sex or gender, race, and/or ethnicity are to be expected. The application or proposal must address plans for the valid analysis of group differences on the basis of sex or gender, race, and ethnicity unless there is clear evidence that such differences are unlikely to be seen.

Progress Reports: For projects involving NIH-defined Phase III Clinical Trials, annual Research Performance Progress Reports (RPPRs) should include a statement indicating the status of analyses of the primary outcome by sex or gender, race, and ethnicity. The results of these analyses should be included in the “Project Outcomes” section of the RPPR. See the Sample Project Outcomes page for an example.

Registering & Reporting in ClinicalTrials.gov: NIH-defined Phase III Clinical Trials that also meet the definition of an applicable clinical trial must report the results of the valid analysis of group differences in ClinicalTrials.gov. The valid analyses should be done for each primary outcome measure by sex or gender, and race and/or ethnicity. Upon study registration in ClinicalTrials.gov, outcome measures should be pre-specified by sex or gender, and race and/or ethnicity to prepare for reporting results in this stratified manner. Refer to the Guidance for Valid Analysis Reporting and NOT-OD-18-014 for additional information.

Policy Notices and Procedures

Upcoming Events

DHSR One pager of resources for external users

• Human Subjects Research
• NIH Office of Research on Women's Health (ORWH)
• National Institute on Minority Health and Health Disparities (NIMHD)
• Diversity and Inclusion in Clinical Trials (NIMHD)
• For NIH Staff

Have additional questions? Contact your program officer or the Inclusion policy team: [email protected]