quantitative research peer reviewed journal article

Quantitative research

Affiliation.

• 1 Faculty of Health and Social Care, University of Hull, Hull, England.
• PMID: 25828021
• DOI: 10.7748/ns.29.31.44.e8681

This article describes the basic tenets of quantitative research. The concepts of dependent and independent variables are addressed and the concept of measurement and its associated issues, such as error, reliability and validity, are explored. Experiments and surveys – the principal research designs in quantitative research – are described and key features explained. The importance of the double-blind randomised controlled trial is emphasised, alongside the importance of longitudinal surveys, as opposed to cross-sectional surveys. Essential features of data storage are covered, with an emphasis on safe, anonymous storage. Finally, the article explores the analysis of quantitative data, considering what may be analysed and the main uses of statistics in analysis.

Keywords: Experiments; measurement; nursing research; quantitative research; reliability; surveys; validity.

• Biomedical Research / methods*
• Double-Blind Method
• Evaluation Studies as Topic
• Longitudinal Studies
• Randomized Controlled Trials as Topic
• United Kingdom

This paper is in the following e-collection/theme issue:

Published on 20.8.2020 in Vol 22 , No 8 (2020) : August

Digital Inequality During a Pandemic: Quantitative Study of Differences in COVID-19–Related Internet Uses and Outcomes Among the General Population

Authors of this article:

Original Paper

• Alexander JAM van Deursen, Prof Dr  

University of Twente, Enschede, Netherlands

Corresponding Author:

Alexander JAM van Deursen, Prof Dr

University of Twente

Drienerlolaan 5

Enschede, 7500AE

Netherlands

Phone: 31 622942142

Fax:31 534893200

Email: [email protected]

Background: The World Health Organization considers coronavirus disease (COVID-19) to be a public emergency threatening global health. During the crisis, the public’s need for web-based information and communication is a subject of focus. Digital inequality research has shown that internet access is not evenly distributed among the general population.

Objective: The aim of this study was to provide a timely understanding of how different people use the internet to meet their information and communication needs and the outcomes they gain from their internet use in relation to the COVID-19 pandemic. We also sought to reveal the extent to which gender, age, personality, health, literacy, education, economic and social resources, internet attitude, material access, internet access, and internet skills remain important factors in obtaining internet outcomes after people engage in the corresponding uses.

Methods: We used a web-based survey to draw upon a sample collected in the Netherlands. We obtained a dataset with 1733 respondents older than 18 years.

Results: Men are more likely to engage in COVID-19–related communication uses. Age is positively related to COVID-19–related information uses and negatively related to information and communication outcomes. Agreeableness is negatively related to both outcomes and to information uses. Neuroticism is positively related to both uses and to communication outcomes. Conscientiousness is not related to any of the uses or outcomes. Introversion is negatively related to communication outcomes. Finally, openness relates positively to all information uses and to both outcomes. Physical health has negative relationships with both outcomes. Health perception contributes positively to information uses and both outcomes. Traditional literacy has a positive relationship with information uses and both outcomes. Education has a positive relationship with information and communication uses. Economic and social resources played no roles. Internet attitude is positively related to information uses and outcomes but negatively related to communication uses and outcomes. Material access and internet access contributed to all uses and outcomes. Finally, several of the indicators and outcomes became insignificant after accounting for engagement in internet uses.

Conclusions: Digital inequality is a major concern among national and international scholars and policy makers. This contribution aimed to provide a broader understanding in the case of a major health pandemic by using the ongoing COVID-19 crisis as a context for empirical work. Several groups of people were identified as vulnerable, such as older people, less educated people, and people with physical health problems, low literacy levels, or low levels of internet skills. Generally, people who are already relatively advantaged are more likely to use the information and communication opportunities provided by the internet to their benefit in a health pandemic, while less advantaged individuals are less likely to benefit. Therefore, the COVID-19 crisis is also enforcing existing inequalities.

Introduction

The World Health Organization considers coronavirus disease (COVID-19) to be a public emergency threatening global health [ 1 ]. Governments worldwide have taken stringent action, including requiring social distancing, closing public services, schools and universities, and canceling cultural events [ 2 , 3 ]. People are being advised or ordered to stay at home and socially isolate themselves to avoid being infected [ 4 ]. The ongoing pandemic represents an outbreak of an unparalleled scale, and it has induced widespread fear and uncertainty.

In this paper, we focus on the role of the internet during the crisis. The internet has become a crucial source for the general public, as it provides access to general information, the latest national and international developments, and guidelines on behavioral norms during the crisis. In this respect, the internet plays an important role in the great challenges facing governments regarding the transfer of knowledge and guidelines to the population at large. When individuals understand the need and rationale behind government-enforced measures, they are more motivated to comply and even adopt measures voluntarily [ 5 , 6 ]. In addition to informational purposes, the internet enables individuals to share news and experiences with people they cannot meet face-to-face, remain in contact with friends and family, seek support, and ask questions of official agencies, including health agencies. Further, the internet enables people to take initiatives such as raising money or preparing packaged meals for people in need, such as health workers or people who have lost their jobs. In sum, the internet plays a vital role for people of all social strata and backgrounds during a time of worldwide crisis. All people should thus be able to use the internet as a source of information and communication.

However, digital inequality research has shown that internet access is not evenly distributed among the general population [ 7 , 8 ]. The basic idea of digital inequality stems from a comparative perspective of social and information inequality, as there are benefits associated with internet access and negative consequences of lack of access [ 9 ]. Calamities are often a story of inequality [ 10 ]; therefore, in this paper, we aimed to gain a deeper and broader understanding of the differences in how people use the internet to cope during the COVID-19 crisis. Van Dijk’s resources and appropriation theory [ 8 ] explains differences or inequalities of internet access by considering personal and positional categories of individuals and the individuals’ resources. Internet access itself is considered to be a process of appropriation involving attitudinal access, material access, skills access, and in the final stage, usage access. The latter entails differences in the type of activities that people perform on the internet. The consequences of the process are the outcomes of internet use. These outcomes in turn reinforce personal and positional inequalities and an unequal distribution of resources [ 8 ] ( Figure 1 ). The first goal of this paper is to provide a timely understanding of how different people use the internet and the outcomes they gain from it in relation to the COVID-19 pandemic.

Internet use and outcome differences between groups of people are likely to have profound consequences on how people manage a crisis. For example, older people are most in danger of being infected with the virus and most likely to die from the infection [ 11 ], and they also use the internet less and have the fewest internet outcomes [ 12 ]. The latter may further endanger their peculiar situation, as limited internet use and outcomes may result in a lack of critical information or necessary support.

COVID-19–Related Internet Uses and Outcomes

To study differences in internet uses and outcomes during the COVID-19 pandemic, it is necessary to understand the types of uses and outcomes that are at play. Typically, uses and outcomes are studied by following conceptual classifications that distinguish different domains, such as economic, social, cultural, or personal domains [ 13 ]. Here, we take the COVID-19 pandemic as the domain of interest. Within this domain, we consider two main and conceptually different types of uses and outcomes: information and communication [ 14 , 15 ]. Information internet uses involve searching for information on all aspects of COVID-19. Potential information outcomes include becoming better informed about the disease, understanding why certain measures are necessary, and limiting the risk of becoming infected by developing greater awareness of one’s own behavior. Communication internet uses include talking to friends about the crisis, asking questions on social media or online fora, giving advice, or offering support to others. Communication outcomes include finding people on the internet who can offer support or share concern, being less lonely, and protecting others from potential COVID-19 risks. Studying both types of uses and outcomes is important, as prior research has shown that communication uses can compensate for information uses to attain beneficial internet outcomes [ 16 ].

Determinants of COVID-19–Related Internet Uses and Outcomes

Digital inequality research suggests that the vast amount of web-based information and communication possibilities around the COVID-19 pandemic are likely to be difficult to grasp and conceptualize for sections of the general population [ 7 ]. Some frequently observed personal categorical inequalities are gender, age, personality, and health [ 7 ]. Earlier research revealed that men and women differ in their internet activities; women are more likely to use email and social media, whereas men are more likely to use the internet to obtain information [ 17 , 18 ]. Age in general has a negative impact on all types of internet uses and outcomes [ 7 ]. In the COVID-19 crisis, older people are especially vulnerable; therefore, it is very important for them to know how to behave and be safe. We hypothesize that (H1) men are more likely to be involved in information-related uses and outcomes while women are more likely to be involved in communication-related internet uses and outcomes regarding COVID-19-related internet uses and outcomes. We also hypothesize that (H2) age contributes negatively to COVID-19–related internet uses and outcomes.

An individual’s personality may hinder or stimulate their engagement in certain COVID-19–related activities. Cognitive appraisal theory suggests that individuals complete two types of cognitive appraisal processes in a crisis [ 19 ]. The process starts with an evaluation of the crisis as a potential source of danger or life disruption. If the crisis is not determined to be dangerous, it is not considered a stressor and does not require intervention. If the crisis is determined to be relevant, it is considered a stressor and must be evaluated further by balancing the demands of the crisis and the person’s resources [ 20 ]. At this point, personality enters the equation [ 20 ]. There is a general consensus regarding the Big Five model when personality traits are studied. This model proposes five personality traits of agreeableness, neuroticism, conscientiousness, introversion, and openness [ 21 ]. However, there is no agreement as to whether these traits contribute to or detract from resisting disturbance [ 20 ]. There is also no consensus on how the Big Five personality traits relate to internet use [ 7 , 22 ]. For example, conscientiousness relates to people who abide by rules. On one hand, one might argue that this would result in a greater need for information on how to behave. On the other hand, the internet is unstructured, and rules and policies are absent to a large extent. When linking personality traits to internet use for psychological adjustments to the COVID-19 crisis, it is not evident whether these traits will support or hinder COVID-19–related internet uses and outcomes. We hypothesize that (H3a) agreeableness, (H3b) neuroticism, (H3c) contentiousness, (H3d) introversion, and (H3e) openness are related to COVID-19–related internet uses and outcomes.

An individual’s health may play an important role in how they approach COVID-19. To gain an elaborate understanding of how health relates to COVID-19–related internet uses and outcomes, we followed earlier research that distinguishes between different health aspects [ 23 ]: A person’s physical functioning or the degree to which their health currently interferes with activities such as sports, carrying groceries, climbing stairs, and walking, their mental health or psychological distress and well-being, and their health perception concerning their own health rating in general. During a crisis, we expect that people with health issues are more likely to turn to the internet for comfort and reassurance. We hypothesize that (H4a) physical functioning, (H4b) mental health, and (H4c) health perception contribute negatively to COVID-19–related internet uses and outcomes.

The final type of personal inequality considered in this study is traditional literacy, which is known to have a substantial impact on how the internet is used [ 24 , 25 ]. We consider literacy to be the ability to read, write, and understand text, which is also framed under the umbrella terms functional literacy or fundamental literacy [ 24 ]. Functional or traditional literacy can be considered as the basic dimension of all literacy concepts [ 26 ]. Considering the crucial role the internet is playing in the COVID-19 crisis, a low level of literacy is a potentially large inhibitor of understanding information and being involved in web-based communication. We hypothesize that (H5) traditional literacy contributes positively to COVID-19–related internet uses and outcomes.

Education is the most observed positional categorial inequality in digital divide research, and it is likely to play a role in the current context. People with higher levels of education are better equipped to comprehend web-based information and benefit from internet use [ 7 ]. We hypothesize that (H6) education contributes positively to COVID-19–related internet uses and outcomes.

When studying differences in internet uses and outcomes, the resources people can access are often derived from Pierre Bourdieu’s capital theory [ 27 ], which stresses the importance of including not only economic but also social and cultural resources to determine one’s status and position in society. In the COVID-19 pandemic, economic and social resources are likely to be important, as earlier research has shown that people with greater economic resources—mostly operationalized as income in digital inequality research—are known to use the internet more efficaciously and productively [ 7 , 28 ]. People with more social resources are more likely to have access to family, friends, or other contacts on the internet [ 29 ]. We hypothesize that (H7a) economic and (H7b) social resources contribute positively to COVID-19–related internet uses and outcomes.

The Internet Appropriation Process

The core of the resources and appropriation theory is access to technology, which is considered as a process of appropriation involving attitudinal, material, skills, and usage access. Attitudinal access concerns a person’s attitude towards the internet; according to theories of technology adoption, this type of access is crucial for using the internet [ 30 ]. Material access can be defined in terms of the different devices that people use to access the internet and all other web-based resources, including desktop computers, laptop computers, tablets, smartphones, game consoles, and interactive televisions [ 31 ]. Skills access concerns the skills necessary to use the internet, ranging from operational and information skills to social and content creation skills [ 32 ]. Prior research has revealed that all three types of internet access directly affect internet uses and outcomes [ 16 ]. We hypothesize that (H8a) attitudinal internet access, (H8b) material internet access, and (H8c) skills internet access contribute positively to COVID-19–related internet uses and outcomes.

The Effects of COVID-19–Related Internet Uses on Their Corresponding Outcomes

A recent multifaceted consideration of digital inequality revealed a strong effect of internet uses on outcomes [ 12 ]. Further, people’s internet activities appeared to be more important than their personal characteristics with regard to inequalities in outcomes of internet use. This suggests that the variables discussed in the prior sections will become less important for obtaining information outcomes when people are involved in COVID-19–related internet information uses. This is also true for COVID-19–related communication uses and outcomes. The second goal of this paper is to reveal the extent to which the indicators discussed remain important for obtaining internet outcomes after people are involved in the corresponding uses.

Recruitment

This study used a web-based survey and drew upon a sample collected in the Netherlands. To obtain a representative sample of the population, we used PanelClix, a professional organization for market research, to provide a panel of approximately 110,000 people. Members of the panel received a small incentive for every survey they completed. In the Netherlands, 98% of the population uses the internet; therefore, the internet user population is very closely representative of the general population in terms of its sociodemographic makeup. The panel included novice and advanced internet users. In total, we aimed to obtain a dataset with approximately 1700 respondents over the age of 18. Eventually, this resulted in the collection of 1733 responses over a 1-week period in April 2020. During the data collection period, three amendments to the sampling frame were made to ensure the representativity of the Dutch population. Accordingly, the analyses revealed that the gender, age, and formal education of our respondents largely matched official census data. As a result, only very small post hoc corrections were needed.

The web-based survey used software that checked for missing responses and prompted users to respond. The survey was pilot-tested with 10 internet users over two rounds. Amendments were made based on the feedback provided. No major comments were provided in the second round. The average time required to complete the survey was 20 minutes.

We initially developed 11 survey items pertaining to COVID-19–related internet use. Respondents were asked to indicate the extent to which they used the internet for various activities in the past month using a 5-point scale (“not” to “multiple times a day”) as an ordinal-level measure. Principal component analysis with varimax rotation was used to determine two underlying usage clusters, one related to information and one to communication. Factor loadings were employed at 0.4 and above for each item [ 33 ]. In total, 8 items (3 for information and 5 for communication) were retained in a two-factor structure with eigenvalues over 1.0, together accounting for 76% of the total variance.

For COVID-19-related information and communication internet outcomes, we developed 14 items mapped onto the use items. A 5-point agreement scale as an ordinal level measure was used. Principal component analysis with varimax rotation resulted in a structure that matched the conceptual definition of information outcomes (4 items) and communication outcomes (4 items). The two factors showed eigenvalues over 1.0 and explained 65% of the variance.

Gender was included as a dichotomous variable, and age was directly asked (mean 50.2, SD 17.0).

Personality was measured with the Quick Big Five personality questionnaire [ 34 ], which consists of 30 adjectives reflecting a valid and reliable measure of the Big Five traits. Participants were asked to rate the extent to which a particular adjective applied to them on a 7-point scale, ranging from completely untrue to completely true. The Cronbach α values for the five traits were .89 for agreeableness, .88 for neuroticism, .88 for conscientiousness, .87 for introversion, and .81 for openness.

Physical health, mental health, and health perception were measured with the Dutch version of the Medical Outcomes Study (MOS) Short-Form General Health Survey (SF-20) [ 35 ]. This instrument enables respondents to assess their general health and generates composite summary scores representing different types of health. We normalized the scales, with higher scores representing better functioning. Physical health was measured with 5 items (2-point scale; α=.89; mean 1.75, SD 0.34), mental health with 5 items (5-point scale; α=.85; mean 3.65, SD 0.77), and health perception with 5 items (5-point scale; α=.86; mean 3.39, SD 0.85).

To measure traditional literacy, we used the validated 11-item Diagnostic Illiteracy Scale [ 36 ]. Sample items included “I have difficulties with reading and understanding information from my municipality” and “I find it difficult to read and understand my telephone bill.” A 5-point agreement scale was used. Scores on the scale exhibited high internal consistency. Items were recoded so that higher scores corresponded with higher levels of literacy (α=.94; mean 4.33, SD 0.71).

To assess education, data regarding degrees earned were collected and used to create three groups: low (primary), middle (secondary), and high (tertiary) educational achievement.

Economic resources were objectively measured by seeking the annual family income in the last 12 months. Twelve categories were recoded into three categories of low for <€30,000 (US $35,503.50), middle for €30,000 to €70,000 (US $35,503.50 to $82841.50), and high for >€70,000 (>US $82841.50). For social resources, we used the MOS Social Support Survey [ 37 ]. Respondents completed 18 items covering emotional support (eg, “Someone you can count on to listen when you need to talk”), informational support (eg, “Someone to give you good advice about a crisis”), and tangible support (eg, “Someone to help you if you were confined to bed”). All items were rated on a 5-point Likert scale with anchors of none of the time (1) and most of the time (5). We computed an aggregate measure of support availability (α=.96; mean 3.83, SD 0.85).

Attitudinal internet access was measured by three items adapted from the Digital Motivation Scale [ 38 ]. A 5-point agreement scale was used, and all items were balanced for the direction of response (α=.74; mean 4.10, SD 0.70). An example statement is “Technologies such as the internet and mobile phones make life easier.” To measure material internet access, we considered 7 devices used to connect to the internet (mean 3.43, SD 1.53). Included were desktop computer, laptop computer, tablet, smartphone, smart TV, game console, and smart device (eg, activity tracker). Finally, skills internet access was adapted from the conceptual idea behind the Internet Skills Scale [ 32 ]. We proposed 30 items reflecting operational, information navigation, social, and creative internet skills. A 20-item single skills construct resulted from the principal component analysis. All items were scored on a 5-point scale that ranged from “not at all true of me” to “very true of me” and exhibited high internal consistency (α=.96; mean 3.67, SD 0.97). Example items are “I know how to open downloaded files,” “I find it hard to decide what the best keywords are to use for online searches,” and “I know which information I should and shouldn’t share online.”

Statistical Analysis

To test the hypotheses and account for the sequentiality between COVID-19–related internet uses and outcomes, hierarchical regression analyses were used. In the first model, we tested our hypotheses by analyzing the significant determinants for the two types of COVID-19–related internet uses and the two corresponding outcomes. In the second model, we sought to determine the changes in the significance of the determinants after the internet uses were added to the models.

Table 1 provides an overview of the sample of people surveyed in the study.

Table 2 shows the mean scores of the survey questions related to internet uses and internet outcomes.

The first goal of this paper was addressed in the first model, as presented in Table 3 , where several significant determinants for COVID-19 uses and outcomes are revealed.

a Low: primary; middle: secondary; high: tertiary.

a COVID-19: coronavirus disease.

b RIVM: Rijksinstituut voor Volksgezondheid en Milieu.

Table 3 shows that men are more likely to be involved in COVID-19–related communication uses. Age is positively related to COVID-19–related information uses and negatively related to COVID-19 communication uses and outcomes. Concerning personality traits, agreeableness is negatively related to COVID-19–related information and communication uses and to communication outcomes. Neuroticism is positively related to both uses and to communication outcomes.

Conscientiousness is not related to any of the uses or outcomes. Introversion is negatively related to COVID-19–related communication uses and outcomes, suggesting that this is performed more by extraverted people. Finally, openness relates positively to information uses and to both outcomes.

The results further show that concerning the three health indicators, physical health is negatively related to communication uses and outcomes. Mental health did not contribute to any uses or outcomes. Health perception contributes positively to information uses and to both outcomes.

Traditional literacy has a positive relationship with information-type uses and with both outcomes, and education has a positive relationship with COVID-19–related information and communication uses. Economic and social resources were not related to any COVID-19 uses or outcomes.

Attitudinal internet access is positively related to information uses and outcomes but is negatively related to communication uses and outcomes. Material internet access contributes positively to all uses and outcomes, and skills access has a positive relationship with all uses and outcomes. Table 4 provides an overview of the hypotheses.

a ns: no significant contribution.

b –: significant negative contribution.

c R: reject.

d +: significant positive contribution.

e PS: partial support.

f S: support.

Finally, to address the second goal of the study, we tested what would happen to the contribution of the outcome determinants when the corresponding uses were added to the analyses (Model 2: see Tables 5 and 6 ). Adding the uses significantly increased the explained variance; also, several of the relationships between personal and positional categories and between resources and outcomes became insignificant. The relationships that remained significant for information outcomes were age, health perception, and traditional literacy. Furthermore, attitudinal internet access remained significant. For communication outcomes, the relationships that remained significant were age, openness, and traditional literacy.

a N/A: not applicable.

Principal Results

This paper aims to provide a comprehensive examination of digital inequality in the case of an unprecedented health pandemic. The first goal of the study was to reveal how inequality manifests itself in COVID-19–related internet information and communication uses and outcomes. The findings revealed several relationships between the background variables and the two types of internet uses and outcomes.

Older people were found to be less equipped to use the internet for information and communication during a time of crisis. However, they were more likely to engage in information-type COVID-19–related internet uses, possibly because they are at greatest risk from the disease [ 11 ]. This did not result in more beneficial information outcomes. Internet skills play an important role in translating internet uses into beneficial internet outcomes [ 39 ], and prior research has shown that older people have lower internet skill levels in general [ 32 ]. The finding that older people are less likely to perform communication activities or obtain communication-related outcomes is in line with prior studies [ 15 ]; however, these outcomes are important, as older people are more at risk of having severe complications when diagnosed with COVID-19. Regarding gender, contrary to general internet use, men were found to be more likely to engage in communication-type COVID-19–related internet uses during the crisis than women. A possible explanation is that men and women may respond to crisis news in different ways [ 40 ].

The positive effect of neuroticism suggests that a tendency to experience negative emotions such as anger, anxiety, or depression fuels the need to turn to the internet for COVID-19–related information and communication. People who score higher on the neuroticism scale may be more in need of guidelines on how to mitigate risks or may need more support from others to be comforted. Also, the openness trait supports both information and communication internet use and outcomes. A possible explanation is that a major crisis triggers adventure, unconventional ideas, imagination, awareness of feelings, curiosity, or a variety of experiences, all of which are aspects linked to high openness [ 21 ]. The negative contribution of agreeableness raises questions. A possible explanation is that agreeable people are less frequently sought out for communication activities. However, the internet may also be a very inviting environment for less agreeable people. Conscientiousness did not appear to be a significant determinant. People who are more stubborn and focused or more flexible and spontaneous both appear to be involved in information- and communication-type COVID-19–related internet uses and outcomes. Extroversion emerged as a trait that supports using the internet for communication uses and outcomes; this can be expected, as extroversion is marked by pronounced engagement with the external world [ 21 ].

Although we expected that psychological distress would play a role in the current context, as there would be a relatively high need for information and support from others, mental health did not surface as a significant contributor. Furthermore, we did find that physical health problems appear to encourage web-based COVID-19–related communication uses and outcomes. The most likely explanation is that people with underlying health problems are more at risk (and thus more bound to their homes) and thus have higher needs for communication with friends and family. A possible reason for the positive effect of health perception is that people who believe their personal health to be good may feel better equipped to support others during the COVID-19 pandemic.

As expected, traditional literacy played an important role. A lack of general ability to read, write, and understand text further disadvantages individuals in the case of the COVID-19 pandemic, as they have less access to information and communication sources. COVID-19 is a new, unknown, and complicated disease with characteristics that are often described in difficult medical language that is not easy to read. Similar findings were found for educational attainment. Research has long shown that education is one of the most prominent positional variables in digital divide research [ 7 ]. However, our results suggest that when less educated individuals are involved in information and communication internet uses, they are as likely to achieve the corresponding outcomes as people with higher levels of education. This is an important finding for designing interventions for those of lower levels of education.

An effect of economic resources did not emerge in relation to COVID-19–related internet uses and outcomes. The participants’ income did not make a difference in obtaining information and communication COVID-19–related internet outcomes. Earlier research often showed that income is especially important to consumptive and work-related internet uses [ 17 ], topics that are not considered here. Unexpectedly, social resources were not found to be influential. Apparently, a person who has an offline support network will not necessarily turn more to web-based information and communication support during a crisis.

Concerning internet access, we can first conclude that a person’s internet attitude is important for engaging in information uses and gaining information outcomes. Unexpectedly, there was a negative contribution of internet attitude to communication uses and outcomes, suggesting that individuals who have a negative evaluation of the internet in general are more likely to engage in communication uses in the event of a major crisis. Both material and skills internet access played important roles in achieving all uses and outcomes. Using a higher diversity of devices was related to higher COVID-19–related internet use and to more outcomes. The opportunities devices offer are known to be related to inequalities in internet uses and outcomes. As each device offers its own specific characteristics and advantages, a higher diversity of devices supports a larger range of use activities and outcomes [ 31 ]. Furthermore, internet skills play a fundamental role in COVID-19–related uses and in obtaining beneficial outcomes [ 12 ].

In this paper, several indicators surfaced for people’s web-based COVID-19–related uses and outcomes. The variety of important indicators raises the question of whether general policies to address digital inequalities in a time of crisis will be effective. The complex relationships between the different indicators on one hand and internet uses and outcomes on the other hand demand more focused policies, such as those related to health indicators and the need for information to enhance health outcomes. This study reveals that the greater an individual’s existing advantages, the more they benefit from the internet at a time of crisis; the converse is true as well. Marginalized people are likely to have fewer types of access available to take actions, behave as requested, or be comforted by help, creating a vicious cycle where already marginalized groups are further marginalized in a time of crisis.

To end on a positive note, the situation may become slightly less complex when we address the second goal of this paper. When people engage in information and communication internet uses in a crisis situation, their personal characteristics become less important to achieving the corresponding outcomes. This suggests that to achieve information and communication outcomes, policy or research should especially focus on encouraging people to engage in the corresponding internet uses, as we can assume to some extent that engagement with information and communication-related COVID-19 uses is the best way to achieve beneficial outcomes at a time when they are most needed.

Limitations

The current study was conducted in the Netherlands, a country whose citizens have very high household internet penetration and high levels of educational attainment. Although differences in educational background and income are present and were taken into consideration, the observed inequalities may be even stronger in countries with a less homogeneous population. Given that the greatest burden of deaths has been in countries with very diverse populations, race and associated factors are likely to play a major role.

The aim of this study was to provide a broader picture of inequality in relation to how the internet is used in the case of a major global health crisis. A broad range of determinants was considered, and the relative importance of these indicators was revealed. However, a deeper understanding and further investigation to reveal the exact underlying mechanisms that cause these indicators to play a role would provide additional explanations. This suggests that further qualitative research is needed not only to obtain in-depth understanding of the mechanisms but also to understand the consequences of the observed inequalities to complement the findings of the current quantitative approach.

Conclusions

Digital inequality is a major concern among national and international scholars and policy makers. In this paper, we aimed to provide a broader understanding in the case of a major health pandemic by using the ongoing COVID-19 crisis as a context for empirical work. Several groups of people were identified as vulnerable, such as older people and people with lower levels of education, physical health problems, higher levels of neuroticism, low literacy levels, and low levels of trust. The general conclusion is that people who are already relatively advantaged are more likely to use the information and communication opportunities provided by the internet to their benefit in a health pandemic, while more disadvantaged individuals are less likely to benefit. Therefore, the COVID-19 crisis is also an enforcer of existing inequalities.

Conflicts of Interest

None declared.

• Mahase E. China coronavirus: WHO declares international emergency as death toll exceeds 200. BMJ 2020 Jan 31;368:m408. [ CrossRef ] [ Medline ]
• Anderson RM, Heesterbeek H, Klinkenberg D, Hollingsworth TD. How will country-based mitigation measures influence the course of the COVID-19 epidemic? Lancet 2020 Mar;395(10228):931-934 [ FREE Full text ] [ CrossRef ]
• Chinazzi M, Davis JT, Ajelli M, Gioannini C, Litvinova M, Merler S, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science 2020 Apr 24;368(6489):395-400 [ FREE Full text ] [ CrossRef ] [ Medline ]
• Horton R. Offline: 2019-nCoV—“A desperate plea”. Lancet 2020 Feb;395(10222):400. [ CrossRef ]
• Cowper A. Covid-19: are we getting the communications right? BMJ 2020 Mar 06;368:m919. [ CrossRef ] [ Medline ]
• Deci E, Ryan R. Intrinsic motivation. In: Weiner I, Craighead WE, editors. The Corsini Encyclopedia Of Psychology. Hoboken, NJ: John Wiley & Sons; 2010:2.
• Scheerder A, van Deursen AJAM, van Dijk JAGM. Determinants of Internet skills, uses and outcomes. A systematic review of the second- and third-level digital divide. Telemat Inform 2017 Dec;34(8):1607-1624. [ CrossRef ]
• Van Dijk JAGM. The Deepening Divide: Inequality in the Information Society. London, UK: Sage Publications; 2005.
• van Deursen AJ, van Dijk JA. The digital divide shifts to differences in usage. New Media Soc 2013 Jun 07;16(3):507-526. [ CrossRef ]
• Madianou M. Digital Inequality and Second-Order Disasters: Social Media in the Typhoon Haiyan Recovery. SM+S 2015 Sep 30;1(2):205630511560338. [ CrossRef ]
• Wang L, He W, Yu X, Hu D, Bao M, Liu H, et al. Coronavirus disease 2019 in elderly patients: Characteristics and prognostic factors based on 4-week follow-up. J Infect 2020 Jun;80(6):639-645 [ FREE Full text ] [ CrossRef ] [ Medline ]
• Van Deursen AJ, Helsper EJ. Collateral benefits of Internet use: Explaining the diverse outcomes of engaging with the Internet. New Media Soc 2018 Jul 30;20(7):2333-2351 [ FREE Full text ] [ CrossRef ] [ Medline ]
• Helsper EJ. A Corresponding Fields Model for the Links Between Social and Digital Exclusion. Commun Theor 2012 Oct 15;22(4):403-426. [ CrossRef ]
• Kraut R, Mukhopadhyay T, Szczypula J, Kiesler S, Scherlis B. Information and Communication: Alternative Uses of the Internet in Households. Inf Sys Res 1999 Dec;10(4):287-303. [ CrossRef ]
• Blank G, Groselj D. Dimensions of Internet use: amount, variety, and types. Information, Communication & Society 2014 Feb 28;17(4):417-435. [ CrossRef ]
• van Deursen AJAM, Courtois C, van Dijk JAGM. Internet Skills, Sources of Support, and Benefiting From Internet Use. International Journal of Human-Computer Interaction 2014 Mar 07;30(4):278-290. [ CrossRef ]
• van Deursen AJ, van Dijk JA, ten Klooster PM. Increasing inequalities in what we do online: A longitudinal cross sectional analysis of Internet activities among the Dutch population (2010 to 2013) over gender, age, education, and income. Telemat Inform 2015 May;32(2):259-272. [ CrossRef ]
• Zillien N, Hargittai E. Digital Distinction: Status‐Specific Types of Internet Usage. Soc Sci Q 2009;90(2):274-291. [ CrossRef ]
• Lazarus R, Folkman S. Cognitive theories of stress and the issue of circularity. In: Appley MH, Trumbull R, editors. Dynamics of Stress. Boston, MA: Spinger; 1986:63-80.
• Lazarus R. Stress and Emotion: A New Synthesis. New York, NY: Springer; 2006:A.
• John O, Srivastava S. The Big 5 trait taxonomy: History, measurement, and theoretical perspectives. In: Pervin LA, John OP, editors. Handbook of Personality: Theory and Research (2nd ed.). New York, NY: Guilford Press; 1999:102-138.
• Landers RN, Lounsbury JW. An investigation of Big Five and narrow personality traits in relation to Internet usage. Comput Hum Behav 2006 Mar;22(2):283-293. [ CrossRef ]
• Stewart AL, Hays RD, Ware JE. The MOS short-form general health survey. Reliability and validity in a patient population. Med Care 1988 Jul;26(7):724-735. [ CrossRef ] [ Medline ]
• Coiro J. Exploring Literacy on the Internet: Reading Comprehension on the Internet: Expanding Our Understanding of Reading Comprehension to Encompass New Literacies. Read Teach 2003;56(5):458-464.
• van Deursen A, van Dijk J. Modeling Traditional Literacy, Internet Skills and Internet Usage: An Empirical Study. Interact Comput 2014 Jul 16;28(1):13-26. [ CrossRef ]
• Frisch A, Camerini L, Diviani N, Schulz PJ. Defining and measuring health literacy: how can we profit from other literacy domains? Health Promot Int 2012 Mar;27(1):117-126. [ CrossRef ] [ Medline ]
• Bourdieu P. The forms of capital. In: Richardson J, editor. Handbook of Theory and Research for the Sociology of Education. New York, NY: Greenwood; 1986:241-258.
• DiMaggio P, Hargittai E, Celeste C, Shafer S. From unequal access to differentiated use: A literature review and agenda for research on digital inequality. In: Neckerman K, editor. Social Inequality. New York: Russell Sage Foundation; 2004:A-400.
• Woolcock M, Narayan D. Social Capital: Implications for Development Theory, Research, and Policy. World Bank Res Obs 2000 Aug 01;15(2):225-249. [ CrossRef ]
• Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q 1989 Sep;13(3):319. [ CrossRef ]
• van Deursen AJ, van Dijk JA. The first-level digital divide shifts from inequalities in physical access to inequalities in material access. New Media Soc 2019 Feb 07;21(2):354-375 [ FREE Full text ] [ CrossRef ] [ Medline ]
• van Deursen AJ, Helsper EJ, Eynon R. Development and validation of the Internet Skills Scale (ISS). Inf Commun Soc 2015 Aug 25;19(6):804-823. [ CrossRef ]
• Field A. Discovering Statistics Using IBM SPSS Statistics. London, UK: Sage Publications; 2013.
• Vermulst AA, Gerris JRM. Quick Big Five Personality Questionnaire. Guideline. Leeuwarden NL: LDC Publications; 2006.
• Kempen GIJM, Brilman EI, Heyink J, Ormel J. MOS Short-Form General Health Survey. Groningen, Netherlands: Rijksuniversiteit Groningen; 1995.
• De Greef M, van Deursen AJAM, Tubbing M. Development of the DIS-scale (Diagnostic Illiteracy Scale) in order to reveal illiteracy among adults. J Study Adult Educ Learn 2013;1:37-48.
• Sherbourne CD, Stewart AL. The MOS social support survey. Soc Sci Med 1991 Jan;32(6):705-714. [ CrossRef ]
• Helsper E, Smirnova S, Robinson D. DiSTO Youth. London School of Economics and Political Science. 2017.   URL: http://www.lse.ac.uk/media-and-communications/research/research-projects/disto/disto-youth [accessed 2020-08-14]
• Van Deursen AJAM, Helsper E, Eynon R, van Dijk JAGM. The compoundness and sequentiality of digital inequality. Int J Commun 2017;11:452-473.
• Lachlan KA, Spence PR, Nelson LD. Gender Differences in Negative Psychological Responses to Crisis News: The Case of the I-35W Collapse. Communication Research Reports 2010 Feb;27(1):38-48. [ CrossRef ]

Abbreviations

Edited by G Eysenbach; submitted 11.05.20; peer-reviewed by S LaValley, T Hale; comments to author 13.07.20; revised version received 16.07.20; accepted 03.08.20; published 20.08.20

©Alexander JAM van Deursen. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 20.08.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.

CONCEPTUAL ANALYSIS article

Quantitative and qualitative approaches to generalization and replication–a representationalist view.

• Foundations of Education, University of Bamberg, Bamberg, Germany

In this paper, we provide a re-interpretation of qualitative and quantitative modeling from a representationalist perspective. In this view, both approaches attempt to construct abstract representations of empirical relational structures. Whereas quantitative research uses variable-based models that abstract from individual cases, qualitative research favors case-based models that abstract from individual characteristics. Variable-based models are usually stated in the form of quantified sentences (scientific laws). This syntactic structure implies that sentences about individual cases are derived using deductive reasoning. In contrast, case-based models are usually stated using context-dependent existential sentences (qualitative statements). This syntactic structure implies that sentences about other cases are justifiable by inductive reasoning. We apply this representationalist perspective to the problems of generalization and replication. Using the analytical framework of modal logic, we argue that the modes of reasoning are often not only applied to the context that has been studied empirically, but also on a between-contexts level. Consequently, quantitative researchers mostly adhere to a top-down strategy of generalization, whereas qualitative researchers usually follow a bottom-up strategy of generalization. Depending on which strategy is employed, the role of replication attempts is very different. In deductive reasoning, replication attempts serve as empirical tests of the underlying theory. Therefore, failed replications imply a faulty theory. From an inductive perspective, however, replication attempts serve to explore the scope of the theory. Consequently, failed replications do not question the theory per se , but help to shape its boundary conditions. We conclude that quantitative research may benefit from a bottom-up generalization strategy as it is employed in most qualitative research programs. Inductive reasoning forces us to think about the boundary conditions of our theories and provides a framework for generalization beyond statistical testing. In this perspective, failed replications are just as informative as successful replications, because they help to explore the scope of our theories.

Introduction

Qualitative and quantitative research strategies have long been treated as opposing paradigms. In recent years, there have been attempts to integrate both strategies. These “mixed methods” approaches treat qualitative and quantitative methodologies as complementary, rather than opposing, strategies ( Creswell, 2015 ). However, whilst acknowledging that both strategies have their benefits, this “integration” remains purely pragmatic. Hence, mixed methods methodology does not provide a conceptual unification of the two approaches.

Lacking a common methodological background, qualitative and quantitative research methodologies have developed rather distinct standards with regard to the aims and scope of empirical science ( Freeman et al., 2007 ). These different standards affect the way researchers handle contradictory empirical findings. For example, many empirical findings in psychology have failed to replicate in recent years ( Klein et al., 2014 ; Open Science, Collaboration, 2015 ). This “replication crisis” has been discussed on statistical, theoretical and social grounds and continues to have a wide impact on quantitative research practices like, for example, open science initiatives, pre-registered studies and a re-evaluation of statistical significance testing ( Everett and Earp, 2015 ; Maxwell et al., 2015 ; Shrout and Rodgers, 2018 ; Trafimow, 2018 ; Wiggins and Chrisopherson, 2019 ).

However, qualitative research seems to be hardly affected by this discussion. In this paper, we argue that the latter is a direct consequence of how the concept of generalizability is conceived in the two approaches. Whereas most of quantitative psychology is committed to a top-down strategy of generalization based on the idea of random sampling from an abstract population, qualitative studies usually rely on a bottom-up strategy of generalization that is grounded in the successive exploration of the field by means of theoretically sampled cases.

Here, we show that a common methodological framework for qualitative and quantitative research methodologies is possible. We accomplish this by introducing a formal description of quantitative and qualitative models from a representationalist perspective: both approaches can be reconstructed as special kinds of representations for empirical relational structures. We then use this framework to analyze the generalization strategies used in the two approaches. These turn out to be logically independent of the type of model. This has wide implications for psychological research. First, a top-down generalization strategy is compatible with a qualitative modeling approach. This implies that mainstream psychology may benefit from qualitative methods when a numerical representation turns out to be difficult or impossible, without the need to commit to a “qualitative” philosophy of science. Second, quantitative research may exploit the bottom-up generalization strategy that is inherent to many qualitative approaches. This offers a new perspective on unsuccessful replications by treating them not as scientific failures, but as a valuable source of information about the scope of a theory.

The Quantitative Strategy–Numbers and Functions

Quantitative science is about finding valid mathematical representations for empirical phenomena. In most cases, these mathematical representations have the form of functional relations between a set of variables. One major challenge of quantitative modeling consists in constructing valid measures for these variables. Formally, to measure a variable means to construct a numerical representation of the underlying empirical relational structure ( Krantz et al., 1971 ). For example, take the behaviors of a group of students in a classroom: “to listen,” “to take notes,” and “to ask critical questions.” One may now ask whether is possible to assign numbers to the students, such that the relations between the assigned numbers are of the same kind as the relations between the values of an underlying variable, like e.g., “engagement.” The observed behaviors in the classroom constitute an empirical relational structure, in the sense that for every student-behavior tuple, one can observe whether it is true or not. These observations can be represented in a person × behavior matrix 1 (compare Figure 1 ). Given this relational structure satisfies certain conditions (i.e., the axioms of a measurement model), one can assign numbers to the students and the behaviors, such that the relations between the numbers resemble the corresponding numerical relations. For example, if there is a unique ordering in the empirical observations with regard to which person shows which behavior, the assigned numbers have to constitute a corresponding unique ordering, as well. Such an ordering coincides with the person × behavior matrix forming a triangle shaped relation and is formally represented by a Guttman scale ( Guttman, 1944 ). There are various measurement models available for different empirical structures ( Suppes et al., 1971 ). In the case of probabilistic relations, Item-Response models may be considered as a special kind of measurement model ( Borsboom, 2005 ).

Figure 1 . Constructing a numerical representation from an empirical relational structure; Due to the unique ordering of persons with regard to behaviors (indicated by the triangular shape of the relation), it is possible to construct a Guttman scale by assigning a number to each of the individuals, representing the number of relevant behaviors shown by the individual. The resulting variable (“engagement”) can then be described by means of statistical analyses, like, e.g., plotting the frequency distribution.

Although essential, measurement is only the first step of quantitative modeling. Consider a slightly richer empirical structure, where we observe three additional behaviors: “to doodle,” “to chat,” and “to play.” Like above, one may ask, whether there is a unique ordering of the students with regard to these behaviors that can be represented by an underlying variable (i.e., whether the matrix forms a Guttman scale). If this is the case, we may assign corresponding numbers to the students and call this variable “distraction.” In our example, such a representation is possible. We can thus assign two numbers to each student, one representing his or her “engagement” and one representing his or her “distraction” (compare Figure 2 ). These measurements can now be used to construct a quantitative model by relating the two variables by a mathematical function. In the simplest case, this may be a linear function. This functional relation constitutes a quantitative model of the empirical relational structure under study (like, e.g., linear regression). Given the model equation and the rules for assigning the numbers (i.e., the instrumentations of the two variables), the set of admissible empirical structures is limited from all possible structures to a rather small subset. This constitutes the empirical content of the model 2 ( Popper, 1935 ).

Figure 2 . Constructing a numerical model from an empirical relational structure; Since there are two distinct classes of behaviors that each form a Guttman scale, it is possible to assign two numbers to each individual, correspondingly. The resulting variables (“engagement” and “distraction”) can then be related by a mathematical function, which is indicated by the scatterplot and red line on the right hand side.

The Qualitative Strategy–Categories and Typologies

The predominant type of analysis in qualitative research consists in category formation. By constructing descriptive systems for empirical phenomena, it is possible to analyze the underlying empirical structure at a higher level of abstraction. The resulting categories (or types) constitute a conceptual frame for the interpretation of the observations. Qualitative researchers differ considerably in the way they collect and analyze data ( Miles et al., 2014 ). However, despite the diverse research strategies followed by different qualitative methodologies, from a formal perspective, most approaches build on some kind of categorization of cases that share some common features. The process of category formation is essential in many qualitative methodologies, like, for example, qualitative content analysis, thematic analysis, grounded theory (see Flick, 2014 for an overview). Sometimes these features are directly observable (like in our classroom example), sometimes they are themselves the result of an interpretative process (e.g., Scheunpflug et al., 2016 ).

In contrast to quantitative methodologies, there have been little attempts to formalize qualitative research strategies (compare, however, Rihoux and Ragin, 2009 ). However, there are several statistical approaches to non-numerical data that deal with constructing abstract categories and establishing relations between these categories ( Agresti, 2013 ). Some of these methods are very similar to qualitative category formation on a conceptual level. For example, cluster analysis groups cases into homogenous categories (clusters) based on their similarity on a distance metric.

Although category formation can be formalized in a mathematically rigorous way ( Ganter and Wille, 1999 ), qualitative research hardly acknowledges these approaches. 3 However, in order to find a common ground with quantitative science, it is certainly helpful to provide a formal interpretation of category systems.

Let us reconsider the above example of students in a classroom. The quantitative strategy was to assign numbers to the students with regard to variables and to relate these variables via a mathematical function. We can analyze the same empirical structure by grouping the behaviors to form abstract categories. If the aim is to construct an empirically valid category system, this grouping is subject to constraints, analogous to those used to specify a measurement model. The first and most important constraint is that the behaviors must form equivalence classes, i.e., within categories, behaviors need to be equivalent, and across categories, they need to be distinct (formally, the relational structure must obey the axioms of an equivalence relation). When objects are grouped into equivalence classes, it is essential to specify the criterion for empirical equivalence. In qualitative methodology, this is sometimes referred to as the tertium comparationis ( Flick, 2014 ). One possible criterion is to group behaviors such that they constitute a set of specific common attributes of a group of people. In our example, we might group the behaviors “to listen,” “to take notes,” and “to doodle,” because these behaviors are common to the cases B, C, and D, and they are also specific for these cases, because no other person shows this particular combination of behaviors. The set of common behaviors then forms an abstract concept (e.g., “moderate distraction”), while the set of persons that show this configuration form a type (e.g., “the silent dreamer”). Formally, this means to identify the maximal rectangles in the underlying empirical relational structure (see Figure 3 ). This procedure is very similar to the way we constructed a Guttman scale, the only difference being that we now use different aspects of the empirical relational structure. 4 In fact, the set of maximal rectangles can be determined by an automated algorithm ( Ganter, 2010 ), just like the dimensionality of an empirical structure can be explored by psychometric scaling methods. Consequently, we can identify the empirical content of a category system or a typology as the set of empirical structures that conforms to it. 5 Whereas the quantitative strategy was to search for scalable sub-matrices and then relate the constructed variables by a mathematical function, the qualitative strategy is to construct an empirical typology by grouping cases based on their specific similarities. These types can then be related to one another by a conceptual model that describes their semantic and empirical overlap (see Figure 3 , right hand side).

Figure 3 . Constructing a conceptual model from an empirical relational structure; Individual behaviors are grouped to form abstract types based on them being shared among a specific subset of the cases. Each type constitutes a set of specific commonalities of a class of individuals (this is indicated by the rectangles on the left hand side). The resulting types (“active learner,” “silent dreamer,” “distracted listener,” and “troublemaker”) can then be related to one another to explicate their semantic and empirical overlap, as indicated by the Venn-diagram on the right hand side.

Variable-Based Models and Case-Based Models

In the previous section, we have argued that qualitative category formation and quantitative measurement can both be characterized as methods to construct abstract representations of empirical relational structures. Instead of focusing on different philosophical approaches to empirical science, we tried to stress the formal similarities between both approaches. However, it is worth also exploring the dissimilarities from a formal perspective.

Following the above analysis, the quantitative approach can be characterized by the use of variable-based models, whereas the qualitative approach is characterized by case-based models ( Ragin, 1987 ). Formally, we can identify the rows of an empirical person × behavior matrix with a person-space, and the columns with a corresponding behavior-space. A variable-based model abstracts from the single individuals in a person-space to describe the structure of behaviors on a population level. A case-based model, on the contrary, abstracts from the single behaviors in a behavior-space to describe individual case configurations on the level of abstract categories (see Table 1 ).

Table 1 . Variable-based models and case-based models.

From a representational perspective, there is no a priori reason to favor one type of model over the other. Both approaches provide different analytical tools to construct an abstract representation of an empirical relational structure. However, since the two modeling approaches make use of different information (person-space vs. behavior-space), this comes with some important implications for the researcher employing one of the two strategies. These are concerned with the role of deductive and inductive reasoning.

In variable-based models, empirical structures are represented by functional relations between variables. These are usually stated as scientific laws ( Carnap, 1928 ). Formally, these laws correspond to logical expressions of the form

In plain text, this means that y is a function of x for all objects i in the relational structure under consideration. For example, in the above example, one may formulate the following law: for all students in the classroom it holds that “distraction” is a monotone decreasing function of “engagement.” Such a law can be used to derive predictions for single individuals by means of logical deduction: if the above law applies to all students in the classroom, it is possible to calculate the expected distraction from a student's engagement. An empirical observation can now be evaluated against this prediction. If the prediction turns out to be false, the law can be refuted based on the principle of falsification ( Popper, 1935 ). If a scientific law repeatedly withstands such empirical tests, it may be considered to be valid with regard to the relational structure under consideration.

In case-based models, there are no laws about a population, because the model does not abstract from the cases but from the observed behaviors. A case-based model describes the underlying structure in terms of existential sentences. Formally, this corresponds to a logical expression of the form

In plain text, this means that there is at least one case i for which the condition XYZ holds. For example, the above category system implies that there is at least one active learner. This is a statement about a singular observation. It is impossible to deduce a statement about another person from an existential sentence like this. Therefore, the strategy of falsification cannot be applied to test the model's validity in a specific context. If one wishes to generalize to other cases, this is accomplished by inductive reasoning, instead. If we observed one person that fulfills the criteria of calling him or her an active learner, we can hypothesize that there may be other persons that are identical to the observed case in this respect. However, we do not arrive at this conclusion by logical deduction, but by induction.

Despite this important distinction, it would be wrong to conclude that variable-based models are intrinsically deductive and case-based models are intrinsically inductive. 6 Both types of reasoning apply to both types of models, but on different levels. Based on a person-space, in a variable-based model one can use deduction to derive statements about individual persons from abstract population laws. There is an analogous way of reasoning for case-based models: because they are based on a behavior space, it is possible to deduce statements about singular behaviors. For example, if we know that Peter is an active learner, we can deduce that he takes notes in the classroom. This kind of deductive reasoning can also be applied on a higher level of abstraction to deduce thematic categories from theoretical assumptions ( Braun and Clarke, 2006 ). Similarly, there is an analog for inductive generalization from the perspective of variable-based modeling: since the laws are only quantified over the person-space, generalizations to other behaviors rely on inductive reasoning. For example, it is plausible to assume that highly engaged students tend to do their homework properly–however, in our example this behavior has never been observed. Hence, in variable-based models we usually generalize to other behaviors by means of induction. This kind of inductive reasoning is very common when empirical results are generalized from the laboratory to other behavioral domains.

Although inductive and deductive reasoning are used in qualitative and quantitative research, it is important to stress the different roles of induction and deduction when models are applied to cases. A variable-based approach implies to draw conclusions about cases by means of logical deduction; a case-based approach implies to draw conclusions about cases by means of inductive reasoning. In the following, we build on this distinction to differentiate between qualitative (bottom-up) and quantitative (top-down) strategies of generalization.

Generalization and the Problem of Replication

We will now extend the formal analysis of quantitative and qualitative approaches to the question of generalization and replicability of empirical findings. For this sake, we have to introduce some concepts of formal logic. Formal logic is concerned with the validity of arguments. It provides conditions to evaluate whether certain sentences (conclusions) can be derived from other sentences (premises). In this context, a theory is nothing but a set of sentences (also called axioms). Formal logic provides tools to derive new sentences that must be true, given the axioms are true ( Smith, 2020 ). These derived sentences are called theorems or, in the context of empirical science, predictions or hypotheses . On the syntactic level, the rules of logic only state how to evaluate the truth of a sentence relative to its premises. Whether or not sentences are actually true, is formally specified by logical semantics.

On the semantic level, formal logic is intrinsically linked to set-theory. For example, a logical statement like “all dogs are mammals,” is true if and only if the set of dogs is a subset of the set of mammals. Similarly, the sentence “all chatting students doodle” is true if and only if the set of chatting students is a subset of the set of doodling students (compare Figure 3 ). Whereas, the first sentence is analytically true due to the way we define the words “dog” and “mammal,” the latter can be either true or false, depending on the relational structure we actually observe. We can thus interpret an empirical relational structure as the truth criterion of a scientific theory. From a logical point of view, this corresponds to the semantics of a theory. As shown above, variable-based and case-based models both give a formal representation of the same kinds of empirical structures. Accordingly, both types of models can be stated as formal theories. In the variable-based approach, this corresponds to a set of scientific laws that are quantified over the members of an abstract population (these are the axioms of the theory). In the case-based approach, this corresponds to a set of abstract existential statements about a specific class of individuals.

In contrast to mathematical axiom systems, empirical theories are usually not considered to be necessarily true. This means that even if we find no evidence against a theory, it is still possible that it is actually wrong. We may know that a theory is valid in some contexts, yet it may fail when applied to a new set of behaviors (e.g., if we use a different instrumentation to measure a variable) or a new population (e.g., if we draw a new sample).

From a logical perspective, the possibility that a theory may turn out to be false stems from the problem of contingency . A statement is contingent, if it is both, possibly true and possibly false. Formally, we introduce two modal operators: □ to designate logical necessity, and ◇ to designate logical possibility. Semantically, these operators are very similar to the existential quantifier, ∃, and the universal quantifier, ∀. Whereas ∃ and ∀ refer to the individual objects within one relational structure, the modal operators □ and ◇ range over so-called possible worlds : a statement is possibly true, if and only if it is true in at least one accessible possible world, and a statement is necessarily true if and only if it is true in every accessible possible world ( Hughes and Cresswell, 1996 ). Logically, possible worlds are mathematical abstractions, each consisting of a relational structure. Taken together, the relational structures of all accessible possible worlds constitute the formal semantics of necessity, possibility and contingency. 7

In the context of an empirical theory, each possible world may be identified with an empirical relational structure like the above classroom example. Given the set of intended applications of a theory (the scope of the theory, one may say), we can now construct possible world semantics for an empirical theory: each intended application of the theory corresponds to a possible world. For example, a quantified sentence like “all chatting students doodle” may be true in one classroom and false in another one. In terms of possible worlds, this would correspond to a statement of contingency: “it is possible that all chatting students doodle in one classroom, and it is possible that they don't in another classroom.” Note that in the above expression, “all students” refers to the students in only one possible world, whereas “it is possible” refers to the fact that there is at least one possible world for each of the specified cases.

To apply these possible world semantics to quantitative research, let us reconsider how generalization to other cases works in variable-based models. Due to the syntactic structure of quantitative laws, we can deduce predictions for singular observations from an expression of the form ∀ i : y i = f ( x i ). Formally, the logical quantifier ∀ ranges only over the objects of the corresponding empirical relational structure (in our example this would refer to the students in the observed classroom). But what if we want to generalize beyond the empirical structure we actually observed? The standard procedure is to assume an infinitely large, abstract population from which a random sample is drawn. Given the truth of the theory, we can deduce predictions about what we may observe in the sample. Since usually we deal with probabilistic models, we can evaluate our theory by means of the conditional probability of the observations, given the theory holds. This concept of conditional probability is the foundation of statistical significance tests ( Hogg et al., 2013 ), as well as Bayesian estimation ( Watanabe, 2018 ). In terms of possible world semantics, the random sampling model implies that all possible worlds (i.e., all intended applications) can be conceived as empirical sub-structures from a greater population structure. For example, the empirical relational structure constituted by the observed behaviors in a classroom would be conceived as a sub-matrix of the population person × behavior matrix. It follows that, if a scientific law is true in the population, it will be true in all possible worlds, i.e., it will be necessarily true. Formally, this corresponds to an expression of the form

The statistical generalization model thus constitutes a top-down strategy for dealing with individual contexts that is analogous to the way variable-based models are applied to individual cases (compare Table 1 ). Consequently, if we apply a variable-based model to a new context and find out that it does not fit the data (i.e., there is a statistically significant deviation from the model predictions), we have reason to doubt the validity of the theory. This is what makes the problem of low replicability so important: we observe that the predictions are wrong in a new study; and because we apply a top-down strategy of generalization to contexts beyond the ones we observed, we see our whole theory at stake.

Qualitative research, on the contrary, follows a different strategy of generalization. Since case-based models are formulated by a set of context-specific existential sentences, there is no need for universal truth or necessity. In contrast to statistical generalization to other cases by means of random sampling from an abstract population, the usual strategy in case-based modeling is to employ a bottom-up strategy of generalization that is analogous to the way case-based models are applied to individual cases. Formally, this may be expressed by stating that the observed qualia exist in at least one possible world, i.e., the theory is possibly true:

This statement is analogous to the way we apply case-based models to individual cases (compare Table 1 ). Consequently, the set of intended applications of the theory does not follow from a sampling model, but from theoretical assumptions about which cases may be similar to the observed cases with respect to certain relevant characteristics. For example, if we observe that certain behaviors occur together in one classroom, following a bottom-up strategy of generalization, we will hypothesize why this might be the case. If we do not replicate this finding in another context, this does not question the model itself, since it was a context-specific theory all along. Instead, we will revise our hypothetical assumptions about why the new context is apparently less similar to the first one than we originally thought. Therefore, if an empirical finding does not replicate, we are more concerned about our understanding of the cases than about the validity of our theory.

Whereas statistical generalization provides us with a formal (and thus somehow more objective) apparatus to evaluate the universal validity of our theories, the bottom-up strategy forces us to think about the class of intended applications on theoretical grounds. This means that we have to ask: what are the boundary conditions of our theory? In the above classroom example, following a bottom-up strategy, we would build on our preliminary understanding of the cases in one context (e.g., a public school) to search for similar and contrasting cases in other contexts (e.g., a private school). We would then re-evaluate our theoretical description of the data and explore what makes cases similar or dissimilar with regard to our theory. This enables us to expand the class of intended applications alongside with the theory.

Of course, none of these strategies is superior per se . Nevertheless, they rely on different assumptions and may thus be more or less adequate in different contexts. The statistical strategy relies on the assumption of a universal population and invariant measurements. This means, we assume that (a) all samples are drawn from the same population and (b) all variables refer to the same behavioral classes. If these assumptions are true, statistical generalization is valid and therefore provides a valuable tool for the testing of empirical theories. The bottom-up strategy of generalization relies on the idea that contexts may be classified as being more or less similar based on characteristics that are not part of the model being evaluated. If such a similarity relation across contexts is feasible, the bottom-up strategy is valid, as well. Depending on the strategy of generalization, replication of empirical research serves two very different purposes. Following the (top-down) principle of generalization by deduction from scientific laws, replications are empirical tests of the theory itself, and failed replications question the theory on a fundamental level. Following the (bottom-up) principle of generalization by induction to similar contexts, replications are a means to explore the boundary conditions of a theory. Consequently, failed replications question the scope of the theory and help to shape the set of intended applications.

We have argued that quantitative and qualitative research are best understood by means of the structure of the employed models. Quantitative science mainly relies on variable-based models and usually employs a top-down strategy of generalization from an abstract population to individual cases. Qualitative science prefers case-based models and usually employs a bottom-up strategy of generalization. We further showed that failed replications have very different implications depending on the underlying strategy of generalization. Whereas in the top-down strategy, replications are used to test the universal validity of a model, in the bottom-up strategy, replications are used to explore the scope of a model. We will now address the implications of this analysis for psychological research with regard to the problem of replicability.

Modern day psychology almost exclusively follows a top-down strategy of generalization. Given the quantitative background of most psychological theories, this is hardly surprising. Following the general structure of variable-based models, the individual case is not the focus of the analysis. Instead, scientific laws are stated on the level of an abstract population. Therefore, when applying the theory to a new context, a statistical sampling model seems to be the natural consequence. However, this is not the only possible strategy. From a logical point of view, there is no reason to assume that a quantitative law like ∀ i : y i = f ( x i ) implies that the law is necessarily true, i.e.,: □(∀ i : y i = f ( x i )). Instead, one might just as well define the scope of the theory following an inductive strategy. 8 Formally, this would correspond to the assumption that the observed law is possibly true, i.e.,: ◇(∀ i : y i = f ( x i )). For example, we may discover a functional relation between “engagement” and “distraction” without referring to an abstract universal population of students. Instead, we may hypothesize under which conditions this functional relation may be valid and use these assumptions to inductively generalize to other cases.

If we take this seriously, this would require us to specify the intended applications of the theory: in which contexts do we expect the theory to hold? Or, equivalently, what are the boundary conditions of the theory? These boundary conditions may be specified either intensionally, i.e., by giving external criteria for contexts being similar enough to the ones already studied to expect a successful application of the theory. Or they may be specified extensionally, by enumerating the contexts where the theory has already been shown to be valid. These boundary conditions need not be restricted to the population we refer to, but include all kinds of contextual factors. Therefore, adopting a bottom-up strategy, we are forced to think about these factors and make them an integral part of our theories.

In fact, there is good reason to believe that bottom-up generalization may be more adequate in many psychological studies. Apart from the pitfalls associated with statistical generalization that have been extensively discussed in recent years (e.g., p-hacking, underpowered studies, publication bias), it is worth reflecting on whether the underlying assumptions are met in a particular context. For example, many samples used in experimental psychology are not randomly drawn from a large population, but are convenience samples. If we use statistical models with non-random samples, we have to assume that the observations vary as if drawn from a random sample. This may indeed be the case for randomized experiments, because all variation between the experimental conditions apart from the independent variable will be random due to the randomization procedure. In this case, a classical significance test may be regarded as an approximation to a randomization test ( Edgington and Onghena, 2007 ). However, if we interpret a significance test as an approximate randomization test, we test not for generalization but for internal validity. Hence, even if we use statistical significance tests when assumptions about random sampling are violated, we still have to use a different strategy of generalization. This issue has been discussed in the context of small-N studies, where variable-based models are applied to very small samples, sometimes consisting of only one individual ( Dugard et al., 2012 ). The bottom-up strategy of generalization that is employed by qualitative researchers, provides such an alternative.

Another important issue in this context is the question of measurement invariance. If we construct a variable-based model in one context, the variables refer to those behaviors that constitute the underlying empirical relational structure. For example, we may construct an abstract measure of “distraction” using the observed behaviors in a certain context. We will then use the term “distraction” as a theoretical term referring to the variable we have just constructed to represent the underlying empirical relational structure. Let us now imagine we apply this theory to a new context. Even if the individuals in our new context are part of the same population, we may still get into trouble if the observed behaviors differ from those used in the original study. How do we know whether these behaviors constitute the same variable? We have to ensure that in any new context, our measures are valid for the variables in our theory. Without a proper measurement model, this will be hard to achieve ( Buntins et al., 2017 ). Again, we are faced with the necessity to think of the boundary conditions of our theories. In which contexts (i.e., for which sets of individuals and behaviors) do we expect our theory to work?

If we follow the rationale of inductive generalization, we can explore the boundary conditions of a theory with every new empirical study. We thus widen the scope of our theory by comparing successful applications in different contexts and unsuccessful applications in similar contexts. This may ultimately lead to a more general theory, maybe even one of universal scope. However, unless we have such a general theory, we might be better off, if we treat unsuccessful replications not as a sign of failure, but as a chance to learn.

Author Contributions

MB conceived the original idea and wrote the first draft of the paper. MS helped to further elaborate and scrutinize the arguments. All authors contributed to the final version of the manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank Annette Scheunpflug for helpful comments on an earlier version of the manuscript.

1. ^ A person × behavior matrix constitutes a very simple relational structure that is common in psychological research. This is why it is chosen here as a minimal example. However, more complex structures are possible, e.g., by relating individuals to behaviors over time, with individuals nested within groups etc. For a systematic overview, compare Coombs (1964) .

2. ^ This notion of empirical content applies only to deterministic models. The empirical content of a probabilistic model consists in the probability distribution over all possible empirical structures.

3. ^ For example, neither the SAGE Handbook of qualitative data analysis edited by Flick (2014) nor the Oxford Handbook of Qualitative Research edited by Leavy (2014) mention formal approaches to category formation.

4. ^ Note also that the described structure is empirically richer than a nominal scale. Therefore, a reduction of qualitative category formation to be a special (and somehow trivial) kind of measurement is not adequate.

5. ^ It is possible to extend this notion of empirical content to the probabilistic case (this would correspond to applying a latent class analysis). But, since qualitative research usually does not rely on formal algorithms (neither deterministic nor probabilistic), there is currently little practical use of such a concept.

6. ^ We do not elaborate on abductive reasoning here, since, given an empirical relational structure, the concept can be applied to both types of models in the same way ( Schurz, 2008 ). One could argue that the underlying relational structure is not given a priori but has to be constructed by the researcher and will itself be influenced by theoretical expectations. Therefore, abductive reasoning may be necessary to establish an empirical relational structure in the first place.

7. ^ We shall not elaborate on the metaphysical meaning of possible worlds here, since we are only concerned with empirical theories [but see Tooley (1999) , for an overview].

8. ^ Of course, this also means that it would be equally reasonable to employ a top-down strategy of generalization using a case-based model by postulating that □(∃ i : XYZ i ). The implications for case-based models are certainly worth exploring, but lie beyond the scope of this article.

Agresti, A. (2013). Categorical Data Analysis, 3rd Edn. Wiley Series In Probability And Statistics . Hoboken, NJ: Wiley.

Google Scholar

Borsboom, D. (2005). Measuring the Mind: Conceptual Issues in Contemporary Psychometrics . Cambridge: Cambridge University Press. doi: 10.1017/CBO9780511490026

CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol . 3, 77–101. doi: 10.1191/1478088706qp063oa

Buntins, M., Buntins, K., and Eggert, F. (2017). Clarifying the concept of validity: from measurement to everyday language. Theory Psychol. 27, 703–710. doi: 10.1177/0959354317702256

Carnap, R. (1928). The Logical Structure of the World . Berkeley, CA: University of California Press.

Coombs, C. H. (1964). A Theory of Data . New York, NY: Wiley.

Creswell, J. W. (2015). A Concise Introduction to Mixed Methods Research . Los Angeles, CA: Sage.

Dugard, P., File, P., and Todman, J. B. (2012). Single-Case and Small-N Experimental Designs: A Practical Guide to Randomization Tests 2nd Edn . New York, NY: Routledge. doi: 10.4324/9780203180938

Edgington, E., and Onghena, P. (2007). Randomization Tests, 4th Edn. Statistics. Hoboken, NJ: CRC Press. doi: 10.1201/9781420011814

Everett, J. A. C., and Earp, B. D. (2015). A tragedy of the (academic) commons: interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Front. Psychol . 6:1152. doi: 10.3389/fpsyg.2015.01152

PubMed Abstract | CrossRef Full Text | Google Scholar

Flick, U. (Ed.). (2014). The Sage Handbook of Qualitative Data Analysis . London: Sage. doi: 10.4135/9781446282243

Freeman, M., Demarrais, K., Preissle, J., Roulston, K., and St. Pierre, E. A. (2007). Standards of evidence in qualitative research: an incitement to discourse. Educ. Res. 36, 25–32. doi: 10.3102/0013189X06298009

Ganter, B. (2010). “Two basic algorithms in concept analysis,” in Lecture Notes In Computer Science. Formal Concept Analysis, Vol. 5986 , eds D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, et al. (Berlin, Heidelberg: Springer Berlin Heidelberg), 312–340. doi: 10.1007/978-3-642-11928-6_22

Ganter, B., and Wille, R. (1999). Formal Concept Analysis . Berlin, Heidelberg: Springer Berlin Heidelberg. doi: 10.1007/978-3-642-59830-2

Guttman, L. (1944). A basis for scaling qualitative data. Am. Sociol. Rev . 9:139. doi: 10.2307/2086306

Hogg, R. V., Mckean, J. W., and Craig, A. T. (2013). Introduction to Mathematical Statistics, 7th Edn . Boston, MA: Pearson.

Hughes, G. E., and Cresswell, M. J. (1996). A New Introduction To Modal Logic . London; New York, NY: Routledge. doi: 10.4324/9780203290644

Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., et al. (2014). Investigating variation in replicability. Soc. Psychol. 45, 142–152. doi: 10.1027/1864-9335/a000178

Krantz, D. H., Luce, D., Suppes, P., and Tversky, A. (1971). Foundations of Measurement Volume I: Additive And Polynomial Representations . New York, NY; London: Academic Press. doi: 10.1016/B978-0-12-425401-5.50011-8

Leavy, P. (2014). The Oxford Handbook of Qualitative Research . New York, NY: Oxford University Press. doi: 10.1093/oxfordhb/9780199811755.001.0001

Maxwell, S. E., Lau, M. Y., and Howard, G. S. (2015). Is psychology suffering from a replication crisis? what does “failure to replicate” really mean? Am. Psychol. 70, 487–498. doi: 10.1037/a0039400

Miles, M. B., Huberman, A. M., and Saldaña, J. (2014). Qualitative Data Analysis: A Methods Sourcebook, 3rd Edn . Los Angeles, CA; London; New Delhi; Singapore; Washington, DC: Sage.

Open Science, Collaboration (2015). Estimating the reproducibility of psychological science. Science 349:Aac4716. doi: 10.1126/science.aac4716

Popper, K. (1935). Logik Der Forschung . Wien: Springer. doi: 10.1007/978-3-7091-4177-9

Ragin, C. (1987). The Comparative Method : Moving Beyond Qualitative and Quantitative Strategies . Berkeley, CA: University Of California Press.

Rihoux, B., and Ragin, C. (2009). Configurational Comparative Methods: Qualitative Comparative Analysis (Qca) And Related Techniques . Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781452226569

Scheunpflug, A., Krogull, S., and Franz, J. (2016). Understanding learning in world society: qualitative reconstructive research in global learning and learning for sustainability. Int. Journal Dev. Educ. Glob. Learn. 7, 6–23. doi: 10.18546/IJDEGL.07.3.02

Schurz, G. (2008). Patterns of abduction. Synthese 164, 201–234. doi: 10.1007/s11229-007-9223-4

Shrout, P. E., and Rodgers, J. L. (2018). Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol . 69, 487–510. doi: 10.1146/annurev-psych-122216-011845

Smith, P. (2020). An Introduction To Formal Logic . Cambridge: Cambridge University Press. doi: 10.1017/9781108328999

Suppes, P., Krantz, D. H., Luce, D., and Tversky, A. (1971). Foundations of Measurement Volume II: Geometrical, Threshold, and Probabilistic Representations . New York, NY; London: Academic Press.

Tooley, M. (Ed.). (1999). Necessity and Possibility. The Metaphysics of Modality . New York, NY; London: Garland Publishing.

Trafimow, D. (2018). An a priori solution to the replication crisis. Philos. Psychol . 31, 1188–1214. doi: 10.1080/09515089.2018.1490707

Watanabe, S. (2018). Mathematical Foundations of Bayesian Statistics. CRC Monographs On Statistics And Applied Probability . Boca Raton, FL: Chapman And Hall.

Wiggins, B. J., and Chrisopherson, C. D. (2019). The replication crisis in psychology: an overview for theoretical and philosophical psychology. J. Theor. Philos. Psychol. 39, 202–217. doi: 10.1037/teo0000137

Keywords: qualitative research, representational measurement, research methodology, modal logic, generalizability, replication crisis

Citation: Borgstede M and Scholz M (2021) Quantitative and Qualitative Approaches to Generalization and Replication–A Representationalist View. Front. Psychol. 12:605191. doi: 10.3389/fpsyg.2021.605191

Received: 11 September 2020; Accepted: 11 January 2021; Published: 05 February 2021.

Reviewed by:

Copyright © 2021 Borgstede and Scholz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Matthias Borgstede, matthias.borgstede@uni-bamberg.de

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Research article
• Open access
• Published: 06 January 2021

Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study

• Aaron J. Harries   ORCID: orcid.org/0000-0001-7107-0995 1 ,
• Carmen Lee 1 ,
• Lee Jones 2 ,
• Robert M. Rodriguez 1 ,
• John A. Davis 2 ,
• Megan Boysen-Osborn 3 ,
• Kathleen J. Kashima 4 ,
• N. Kevin Krane 5 ,
• Guenevere Rae 6 ,
• Nicholas Kman 7 ,
• Jodi M. Langsfeld 8 &
• Marianne Juarez 1  

BMC Medical Education volume  21 , Article number:  14 ( 2021 ) Cite this article

155k Accesses

169 Citations

38 Altmetric

Metrics details

The COVID-19 pandemic disrupted the United States (US) medical education system with the necessary, yet unprecedented Association of American Medical Colleges (AAMC) national recommendation to pause all student clinical rotations with in-person patient care. This study is a quantitative analysis investigating the educational and psychological effects of the pandemic on US medical students and their reactions to the AAMC recommendation in order to inform medical education policy.

The authors sent a cross-sectional survey via email to medical students in their clinical training years at six medical schools during the initial peak phase of the COVID-19 pandemic. Survey questions aimed to evaluate students’ perceptions of COVID-19’s impact on medical education; ethical obligations during a pandemic; infection risk; anxiety and burnout; willingness and needed preparations to return to clinical rotations.

Seven hundred forty-one (29.5%) students responded. Nearly all students (93.7%) were not involved in clinical rotations with in-person patient contact at the time the study was conducted. Reactions to being removed were mixed, with 75.8% feeling this was appropriate, 34.7% guilty, 33.5% disappointed, and 27.0% relieved.

Most students (74.7%) agreed the pandemic had significantly disrupted their medical education, and believed they should continue with normal clinical rotations during this pandemic (61.3%). When asked if they would accept the risk of infection with COVID-19 if they returned to the clinical setting, 83.4% agreed.

Students reported the pandemic had moderate effects on their stress and anxiety levels with 84.1% of respondents feeling at least somewhat anxious. Adequate personal protective equipment (PPE) (53.5%) was the most important factor to feel safe returning to clinical rotations, followed by adequate testing for infection (19.3%) and antibody testing (16.2%).

Conclusions

The COVID-19 pandemic disrupted the education of US medical students in their clinical training years. The majority of students wanted to return to clinical rotations and were willing to accept the risk of COVID-19 infection. Students were most concerned with having enough PPE if allowed to return to clinical activities.

Peer Review reports

The COVID-19 pandemic has tested the limits of healthcare systems and challenged conventional practices in medical education. The rapid evolution of the pandemic dictated that critical decisions regarding the training of medical students in the United States (US) be made expeditiously, without significant input or guidance from the students themselves. On March 17, 2020, for the first time in modern US history, the Association of American Medical Colleges (AAMC), the largest national governing body of US medical schools, released guidance recommending that medical students immediately pause all clinical rotations to allow time to obtain additional information about the risks of COVID-19 and prepare for safe participation in the future. This decisive action would also conserve scarce resources such as personal protective equipment (PPE) and testing kits; minimize exposure of healthcare workers (HCWs) and the general population; and protect students’ education and wellbeing [ 1 ].

A similar precedent was set outside of the US during the SARS-CoV1 epidemic in 2003, where an initial cluster of infection in medical students in Hong Kong resulted in students being removed from hospital systems where SARS surfaced, including Hong Kong, Singapore and Toronto [ 2 , 3 ]. Later, studies demonstrated that the exclusion of Canadian students from those clinical environments resulted in frustration at lost learning opportunities and students’ inability to help [ 3 ]. International evidence also suggests that medical students perceive an ethical obligation to participate in pandemic response, and are willing to participate in scenarios similar to the current COVID-19 crisis, even when they believe the risk of infection to themselves to be high [ 4 , 5 , 6 ].

The sudden removal of some US medical students from educational settings has occurred previously in the wake of local disasters, with significant academic and personal impacts. In 2005, it was estimated that one-third of medical students experienced some degree of depression or post-traumatic stress disorder (PTSD) after Hurricane Katrina resulted in the closure of Tulane University School of Medicine [ 7 ].

Prior to the current COVID-19 pandemic, we found no studies investigating the effects of pandemics on the US medical education system or its students. The limited pool of evidence on medical student perceptions comes from two earlier global coronavirus surges, SARS and MERS, and studies of student anxiety related to pandemics are also limited to non-US populations [ 3 , 8 , 9 ]. Given the unprecedented nature of the current COVID-19 pandemic, there is concern that students may be missing out on meaningful educational experiences and months of clinical training with unknown effects on their current well-being or professional trajectory [ 10 ].

Our study, conducted during the initial peak phase of the COVID-19 pandemic, reports students’ perceptions of COVID-19’s impact on: medical student education; ethical obligations during a pandemic; perceptions of infection risk; anxiety and burnout; willingness to return to clinical rotations; and needed preparations to return safely. This data may help inform policies regarding the roles of medical students in clinical training during the current pandemic and prepare for the possibility of future pandemics.

We conducted a cross-sectional survey during the initial peak phase of the COVID-19 pandemic in the United States, from 4/20/20 to 5/25/20, via email sent to all clinically rotating medical students at six US medical schools: University of California San Francisco School of Medicine (San Francisco, CA), University of California Irvine School of Medicine (Irvine, CA), Tulane University School of Medicine (New Orleans, LA), University of Illinois College of Medicine (Chicago, Peoria, Rockford, and Urbana, IL), Ohio State University College of Medicine (Columbus, OH), and Zucker School of Medicine at Hofstra/Northwell (Hempstead, NY). Traditional undergraduate medical education in the US comprises 4 years of medical school with 2 years of primarily pre-clinical classroom learning followed by 2 years of clinical training involving direct patient care. Study participants were defined as medical students involved in their clinical training years at whom the AAMC guidance statement was directed. Depending on the curricular schedule of each medical school, this included intended graduation class years of 2020 (graduating 4th year student), 2021 (rising 4th year student), and 2022 (rising 3rd year student), exclusive of planned time off. Participating schools were specifically chosen to represent a broad spectrum of students from different regions of the country (West, South, Midwest, East) with variable COVID-19 prevalence. We excluded medical students not yet involved in clinical rotations. This study was deemed exempt by the respective Institutional Review Boards.

We developed a survey instrument modeled after a survey used in a previously published peer reviewed study evaluating the effects of the COVID-19 pandemic on Emergency Physicians, which incorporated items from validated stress scales [ 11 ]. The survey was modified for use in medical students to assess perceptions of the following domains: perceived impact on medical student education; ethical beliefs surrounding obligations to participate clinically during the pandemic; perceptions of personal infection risk; anxiety and burnout related to the pandemic; willingness to return to clinical rotations; and preparation needed for students to feel safe in the clinical environment. Once created, the survey underwent an iterative process of input and review from our team of authors with experience in survey methodology and psychometric measures to allow for optimization of content and validity. We tested a pilot of our preliminary instrument on five medical students to ensure question clarity, and confirm completion of the survey in approximately 10 min. The final survey consisted of 29 Likert, yes/no, multiple choice, and free response questions. Both medical school deans and student class representatives distributed the survey via email, with three follow-up emails to increase response rates. Data was collected anonymously.

For example, to assess the impact on students’ anxiety, participants were asked, “How much has the COVID-19 pandemic affected your stress or anxiety levels?” using a unipolar 7-point scale (1 = not at all, 4 = somewhat, 7 = extremely). To assess willingness to return to clinical rotations, participants were asked to rate on a bipolar scale (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = neither disagree nor agree, 5 = somewhat agree, 6 = agree, and 7 = strongly agree) their agreement with the statement: “to the extent possible, medical students should continue with normal clinical rotations during this pandemic.” (Survey Instrument, Supplemental Table  1 ).

Survey data was managed using Qualtrics hosted by the University of California, San Francisco. For data analysis we used STATA v15.1 (Stata Corp, College Station, TX). We summarized respondent characteristics and key responses as raw counts, frequency percent, medians and interquartile ranges (IQR). For responses to bipolar questions, we combined positive responses (somewhat agree, agree, or strongly agree) into an agreement percentage. To compare differences in medians we used a signed rank test with p value < 0.05 to show statistical difference. In a secondary analysis we stratified data to compare questions within key domains amongst the following sub-groups: female versus male, graduation year, local community COVID-19 prevalence (high, medium, low), and students on clinical rotations with in-person patient care. This secondary analysis used a chi square test with p value < 0.05 to show statistical difference between sub-group agreement percentages.

Of 2511 students contacted, we received 741 responses (29.5% response rate). Of these, 63.9% of respondents were female and 35.1% were male, with 1.0% reporting a different gender identity; 27.7% of responses came from the class of 2020, 53.5% from the class of 2021, and 18.7% from the class of 2022. (Demographics, Table 1 ).

Most student respondents (74.9%) had a clinical rotation that was cut short or canceled due to COVID-19 and 93.7% reported not being involved in clinical rotations with in-person patient contact at the time of the study. Regarding students’ perceptions of cancelled rotations (allowing for multiple reactions), 75.8% felt this was appropriate, 34.7% felt guilty for not being able to help patients and colleagues, 33.5% felt disappointed, and 27.0% felt relieved.

Most students (74.7%) agreed that their medical education had been significantly disrupted by the pandemic. Students also felt they were able to find meaningful learning experiences during the pandemic (72.1%). Free response examples included: taking a novel COVID-19 pandemic elective course, telehealth patient care, clinical rotations transitioned to virtual online courses, research or education electives, clinical and non-clinical COVID-19-related volunteering, and self-guided independent study electives. Students felt their medical schools were doing everything they could to help students adjust (72.7%). Overall, respondents felt the pandemic had interfered with their ability to develop skills needed to prepare for residency (61.4%), though fewer (45.7%) felt it had interfered with their ability to apply to residency. (Educational Impact, Fig.  1 ).

Perceived educational impacts of the COVID-19 pandemic on medical students

A majority of medical students agreed they should be allowed to continue with normal clinical rotations during this pandemic (61.3%). Most students agreed (83.4%) that they accepted the risk of being infected with COVID-19, if they returned. When asked if students should be allowed to volunteer in clinical settings even if there is not a healthcare worker (HCW) shortage, 63.5% agreed; however, in the case of a HCW shortage only 19.5% believed students should be required to volunteer clinically. (Willingness to Participate Clinically, Fig.  2 ).

Willingness to participate clinically during the COVID-19 pandemic

When asked if they perceived a moral, ethical, or professional obligation for medical students to help, 37.8% agreed that medical students have such an obligation during the current pandemic. This is in contrast to their perceptions of physicians: 87.1% of students agreed with a physician obligation to help during the COVID-19 pandemic. For both groups, students were asked if this obligation persisted without adequate PPE: only 10.9% of students believed medical students had this obligation, while 34.0% agreed physicians had this obligation. (Ethical Obligation, Fig.  3 ).

Ethical obligation to volunteer during the COVID-19 pandemic

Given the assumption that there will not be a COVID-19 vaccine until 2021, students felt the single most important factor in a safe return to clinical rotations was having access to adequate PPE (53.3%), followed by adequate testing for infection (19.3%) and antibody testing for possible immunity (16.2%). Few students (5%) stated that nothing would make them feel comfortable until a vaccine is available. On a 1–7 scale (1 = not at all, 4 = somewhat, 7 = extremely), students felt somewhat prepared to use PPE during this pandemic in the clinical setting, median = 4 (IQR 4,6), and somewhat confident identifying symptoms most concerning for COVID-19, median = 4 (IQR 4,5). Students preferred to learn about PPE via video demonstration (76.7%), online modules (47.7%), and in-person or Zoom style conferences (44.7%).

Students believed they were likely to contract COVID-19 in general (75.6%), independent of a return to the clinical environment. Most respondents believed that missing some school or work would be a likely outcome (90.5%), and only a minority of students believed that hospitalization (22.1%) or death (4.3%) was slightly, moderately, or extremely likely.

On a 1–7 scale (1 = not at all, 4 = somewhat, and 7 = extremely), the median (IQR) reported effect of the COVID-19 pandemic on students’ stress or anxiety level was 5 (4, 6) with 84.1% of respondents feeling at least somewhat anxious due to the pandemic. Students’ perceived emotional exhaustion and burnout before the pandemic was a median = 2 (IQR 2,4) and since the pandemic started a median = 4 (IQR 2,5) with a median difference Δ = 2, p value < 0.001.

Secondary analysis of key questions revealed statistical differences between sub-groups. Women were significantly more likely than men to agree that the pandemic had affected their anxiety. Several significant differences existed for the class of 2020 when compared to the classes of 2021 and 2022: they were less likely to report disruptions to their education, to prefer to return to rotations, and to report an effect on anxiety. There were no significant differences with students who were still involved with in-person patient care compared with those who were not. In comparing areas with high COVID-19 prevalence at the time of the survey (New York and Louisiana) with medium (Illinois and Ohio) and low prevalence (California), students were less likely to report that the pandemic had disrupted their education. Students in low prevalence areas were most likely to agree that medical students should return to rotations. There were no differences between prevalence groups in accepting the risk of infection to return, or subjective anxiety effects. (Stratification, Table  2 ).

The COVID-19 pandemic has fundamentally transformed education at all levels - from preschool to postgraduate. Although changes to K-12 and college education have been well documented [ 12 , 13 ], there have been very few studies to date investigating the effects of COVID-19 on undergraduate medical education [ 14 ]. To maintain the delicate balance between student safety and wellbeing, and the time-sensitive need to train future physicians, student input must guide decisions regarding their roles in the clinical arena. Student concerns related to the pandemic, paired with their desire to return to rotations despite the risks, suggest that medical students may take on emotional burdens as members of the patient care team even when not present in the clinical environment. This study offers insight into how best to support medical students as they return to clinical rotations, how to prepare them for successful careers ahead, and how to plan for their potential roles in future pandemics.

Previous international studies of medical student attitudes towards hypothetical influenza-like pandemics demonstrated a willingness (80%) [ 4 ] and a perceived ethical obligation to volunteer (77 and 70%), despite 40% of Canadian students in one study perceiving a high likelihood of becoming infected [ 5 , 6 ]. Amidst the current COVID-19 pandemic, our participants reported less agreement with a medical student ethical obligation to volunteer in the clinical setting at 37.8%, but believed in a higher likelihood of becoming infected at 75.6%. Their willingness to be allowed to volunteer freely (63.5%) may suggest that the stresses of an ongoing pandemic alter students’ perceptions of the ethical requirement more than their willingness to help. Students overwhelmingly agreed that physicians had an ethical obligation to provide care during the COVID-19 pandemic (87.1%), possibly reflecting how they view the ethical transition from student to physician, or differences between paid professionals and paying for an education.

At the time our study was conducted, there were widespread concerns for possible HCW shortages. It was unclear whether medical students would be called to volunteer when residents became ill, or even graduate early to start residency training immediately (as occurred at half of schools surveyed). This timing allowed us to capture a truly unique perspective amongst medical students, a majority of whom reported increased anxiety and burnout due to the pandemic. At the same time, students felt that their medical schools were doing everything possible to support them, perhaps driven by virtual town halls and daily communication updates.

Trends in secondary analysis show important differences in the impacts of the pandemic. Women were more likely to report increased anxiety as compared to men, which may reflect broader gender differences in medical student anxiety [ 15 ] but requires more study to rule out different pandemic stresses by gender. Graduating medical students (class of 2020) overall described less impact on medical education and anxiety, a decreased desire to return to rotations, but equal acceptance of the risk of infection in clinical settings, possibly reflecting a focus on their upcoming intern year rather than the remaining months of undergraduate medical education. Since this class’s responses decreased overall agreement on these questions, educational impacts and anxiety effects may have been even greater had they been assessed further from graduation. Interestingly, students from areas with high local COVID-19 prevalence (New York and Louisiana) reported a less significant effect of the pandemic on their education, a paradoxical result that may indicate that medical student tolerance for the disruptions was greater in high-prevalence areas, as these students were removed at the same, if not higher, rates as their peers. Our results suggest that in future waves of the current pandemic or other disasters, students may be more patient with educational impacts when they have more immediate awareness of strains on the healthcare system.

A limitation of our study was the survey response rate, which was anticipated given the challenges students were facing. Some may not have been living near campus; others may have stopped reading emails due to early graduation or limited access to email; and some would likely be dealing with additional personal challenges related to the pandemic. We attempted to increase response rates by having the study sent directly from medical school deans and leadership, as well as respective class representatives, and by sending reminders for completion. The survey was not incentivized, and a higher response rate in the class of 2021 across all schools may indicate that students who felt their education was most affected were most likely to respond. We addressed this potential source of bias in the secondary analysis, which showed no differences between 2021 and 2022 respondents. Another limitation was the inherent issue with survey data collection of missing responses for some questions that occurred in a small number of surveys. This resulted in slight variability in the total responses received for certain questions, which were not statistically significant. To be transparent about this limitation, we presented our data by stating each total response and denominator in the Tables.

This initial study lays the groundwork for future investigations and next steps. With 72.1% of students agreeing that they were able to find meaningful learning in spite of the pandemic, future research should investigate novel learning modalities that were successful during this time. Educators should consider additional training on PPE use, given only moderate levels of student comfort in this area, which may be best received via video. It is also important to study the long-term effects of missing several months of essential clinical training and identifying competencies that may not have been achieved, since students perceived a significant disruption to their ability to prepare skills for residency. Next steps could be to study curriculum interventions, such as capstone boot camps and targeted didactic skills training, to help students feel more comfortable as they transition into residency. Educators must also acknowledge that some students may not feel comfortable returning to the clinical environment until a vaccine becomes available (5%) and ensure they are equally supported. Lastly, it is vital to further investigate the mental health effects of the pandemic on medical students, identifying subgroups with additional stressors, needs related to anxiety or possible PTSD, and ways to minimize these negative effects.

In this cross-sectional survey, conducted during the initial peak phase of the COVID-19 pandemic, we capture a snapshot of the effects of the pandemic on US medical students and gain insight into their reactions to the unprecedented AAMC national recommendation for removal from clinical rotations. Student respondents from across the US similarly recognized a significant disruption to their medical education, shared a desire to continue with in-person rotations, and were willing to accept the risk of infection with COVID-19. Our novel results provide a solid foundation to help shape medical student roles in the clinical environment during this pandemic and future outbreaks.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Association of American Medical Colleges. Interim Guidance on Medical Students’ Participation in Direct Patient Contact Activities: Principles and Guidelines. https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak . Published March 17, 2020. Accessed April 1, 2020.

Clark J. Fear of SARS thwarts medical education in Toronto. BMJ. 2003;326(7393):784. https://doi.org/10.1136/bmj.326.7393.784/c .

Article   Google Scholar  

Loh LC, Ali AM, Ang TH, Chelliah A. Impact of a spreading epidemic on medical students. Malays J Med Sci. 2006;13(2):30–6.

Google Scholar  

Mortelmans LJ, Bouman SJ, Gaakeer MI, Dieltiens G, Anseeuw K, Sabbe MB. Dutch senior medical students and disaster medicine: a national survey. Int J Emerg Med. 2015;8(1):77. https://doi.org/10.1186/s12245-015-0077-0 .

Huapaya JA, Maquera-Afaray J, García PJ, Cárcamo C, Cieza JA. Conocimientos, prácticas y actitudes hacia el voluntariado ante una influenza pandémica: estudio transversal con estudiantes de medicina en Perú [Knowledge, practices and attitudes toward volunteer work in an influenza pandemic: cross-sectional study with Peruvian medical students]. Medwave. 2015;15(4):e6136Published 2015 May 8. https://doi.org/10.5867/medwave.2015.04.6136 .

Herman B, Rosychuk RJ, Bailey T, Lake R, Yonge O, Marrie TJ. Medical students and pandemic influenza. Emerg Infect Dis. 2007;13(11):1781–3. https://doi.org/10.3201/eid1311.070279 .

Kahn MJ, Markert RJ, Johnson JE, Owens D, Krane NK. Psychiatric issues and answers following hurricane Katrina. Acad Psychiatry. 2007;31(3):200–4. https://doi.org/10.1176/appi.ap.31.3.200 .

Al-Rabiaah A, Temsah MH, Al-Eyadhy AA, et al. Middle East respiratory syndrome-Corona virus (MERS-CoV) associated stress among medical students at a university teaching hospital in Saudi Arabia. J Infect Public Health. 2020;13(5):687–91. https://doi.org/10.1016/j.jiph.2020.01.005 .

Wong JG, Cheung EP, Cheung V, et al. Psychological responses to the SARS outbreak in healthcare students in Hong Kong. Med Teach. 2004;26(7):657–9. https://doi.org/10.1080/01421590400006572 .

Stokes DC. Senior medical students in the COVID-19 response: an opportunity to be proactive. Acad Emerg Med. 2020;27(4):343–5. https://doi.org/10.1111/acem.13972 .

Rodriguez RM, Medak AJ, Baumann BM, et al. Academic emergency medicine physicians’ anxiety levels, stressors, and potential mitigation measures during the acceleration phase of the COVID-19 pandemic. Acad Emerg Med. 2020;27(8):700–7. https://doi.org/10.1111/acem.14065 .

Sahu P. Closure of universities due to coronavirus disease 2019 (COVID-19): impact on education and mental health of students and academic staff. Cureus. 2020;12(4):e7541Published 2020 Apr 4. https://doi.org/10.7759/cureus.7541 .

Reimers FM, Schleicher A. A framework to guide an education response to the COVID-19 pandemic of 2020: OECD. https://www.hm.ee/sites/default/files/framework_guide_v1_002_harward.pdf .

Choi B, Jegatheeswaran L, Minocha A, Alhilani M, Nakhoul M, Mutengesa E. The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC Med Educ. 2020;20:206–16. https://doi.org/10.1186/s12909-020-02117-1 .

Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian medical students. Acad Med. 2006;81(4):354–73. https://doi.org/10.1097/00001888-200604000-00009 .

Download references

Acknowledgments

The authors wish to thank Newton Addo, UCSF Statistician.

Author information

Authors and affiliations.

Department of Emergency Medicine, University of California San Francisco School of Medicine, San Francisco General Hospital, 1001 Potrero Avenue, Building 5, Room #6A4, San Francisco, California, 94110, USA

Aaron J. Harries, Carmen Lee, Robert M. Rodriguez & Marianne Juarez

University of California San Francisco School of Medicine, San Francisco, California, USA

Lee Jones & John A. Davis

Clinical Emergency Medicine, University of California Irvine School of Medicine, Irvine, CA, USA

Megan Boysen-Osborn

University of Illinois College of Medicine, Chicago, IL, USA

Kathleen J. Kashima

Deming Department of Medicine, Tulane University School of Medicine, New Orleans, Louisiana, USA

N. Kevin Krane

Basic Science Education, Tulane University School of Medicine, New Orleans, Louisiana, USA

Guenevere Rae

Emergency Medicine, Ohio State College of Medicine, Columbus, OH, USA

Nicholas Kman

Department of Science Education, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York, USA

Jodi M. Langsfeld

You can also search for this author in PubMed   Google Scholar

Contributions

All authors made substantial contributions to the study and met the specific conditions listed in the BMC Medical Education editorial policy for authorship. All authors have read and approved the manuscript. AH as principal investigator contributed to study design, survey instrument creation, IRB submission for his respective medical school, acquisition of data and recruitment of other participating medical schools, data analysis, writing and editing the manuscript. CL contributed to background literature review, study design, survey instrument creation, acquisition of data, data analysis, writing and editing the manuscript. LJ contributed to study design, survey instrument creation, acquisition of data from his respective medical school and recruitment of other participating medical schools, data analysis, and editing the manuscript. RR contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript. JD contributed to study design, survey instrument creation, recruitment of other participating medical schools, data analysis, and editing the manuscript. MBO contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. KK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NKK contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. GR contributed as individual site co-principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. NK contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from his respective medical school along with editing the manuscript. JL contributed as individual site principal investigator obtaining IRB exemption acceptance and acquisition of data from her respective medical school along with editing the manuscript. MJ contributed to study design, survey instrument creation, data analysis, writing and editing the manuscript.

Corresponding authors

Correspondence to Aaron J. Harries or Marianne Juarez .

Ethics declarations

Ethics approval and consent to participate.

This study was reviewed and deemed exempt by each participating medical school’s Institutional Review Board (IRB): University of California San Francisco School of Medicine, IRB# 20–30712, Reference# 280106, Tulane University School of Medicine, Reference # 2020–331, University of Illinois College of Medicine), IRB Protocol # 2012–0783, Ohio State University College of Medicine, Study ID# 2020E0463, Zucker School of Medicine at Hofstra/Northwell, Reference # 20200527-SOM-LAN-1, University of California Irvine School of Medicine, submitted self-exemption IRB form. In accordance with the IRB exemption approval, each survey participant received an email consent describing the study and their optional participation.

Consent for publication

This manuscript does not contain any individualized person’s data, therefore consent for publication was not necessary according to the IRB exemption approval.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Survey Instrument

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Harries, A.J., Lee, C., Jones, L. et al. Effects of the COVID-19 pandemic on medical students: a multicenter quantitative study. BMC Med Educ 21 , 14 (2021). https://doi.org/10.1186/s12909-020-02462-1

Download citation

Received : 29 July 2020

Accepted : 16 December 2020

Published : 06 January 2021

DOI : https://doi.org/10.1186/s12909-020-02462-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Undergraduate medical education
• COVID-19 pandemic
• Medical student anxiety

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

Quantitative Research

• Reference work entry
• First Online: 13 January 2019
• Cite this reference work entry

• Leigh A. Wilson 2 , 3  

5419 Accesses

4 Citations

Quantitative research methods are concerned with the planning, design, and implementation of strategies to collect and analyze data. Descartes, the seventeenth-century philosopher, suggested that how the results are achieved is often more important than the results themselves, as the journey taken along the research path is a journey of discovery. High-quality quantitative research is characterized by the attention given to the methods and the reliability of the tools used to collect the data. The ability to critique research in a systematic way is an essential component of a health professional’s role in order to deliver high quality, evidence-based healthcare. This chapter is intended to provide a simple overview of the way new researchers and health practitioners can understand and employ quantitative methods. The chapter offers practical, realistic guidance in a learner-friendly way and uses a logical sequence to understand the process of hypothesis development, study design, data collection and handling, and finally data analysis and interpretation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Writing Quantitative Research Studies

Qualitative Research Methods

Babbie ER. The practice of social research. 14th ed. Belmont: Wadsworth Cengage; 2016.

Google Scholar  

Descartes. Cited in Halverston, W. (1976). In: A concise introduction to philosophy, 3rd ed. New York: Random House; 1637.

Doll R, Hill AB. The mortality of doctors in relation to their smoking habits. BMJ. 1954;328(7455):1529–33. https://doi.org/10.1136/bmj.328.7455.1529 .

Article   Google Scholar  

Liamputtong P. Research methods in health: foundations for evidence-based practice. 3rd ed. Melbourne: Oxford University Press; 2017.

McNabb DE. Research methods in public administration and nonprofit management: quantitative and qualitative approaches. 2nd ed. New York: Armonk; 2007.

Merriam-Webster. Dictionary. http://www.merriam-webster.com . Accessed 20th December 2017.

Olesen Larsen P, von Ins M. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics. 2010;84(3):575–603.

Pannucci CJ, Wilkins EG. Identifying and avoiding bias in research. Plast Reconstr Surg. 2010;126(2):619–25. https://doi.org/10.1097/PRS.0b013e3181de24bc .

Petrie A, Sabin C. Medical statistics at a glance. 2nd ed. London: Blackwell Publishing; 2005.

Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 3rd ed. New Jersey: Pearson Publishing; 2009.

Sheehan J. Aspects of research methodology. Nurse Educ Today. 1986;6:193–203.

Wilson LA, Black DA. Health, science research and research methods. Sydney: McGraw Hill; 2013.

Download references

Author information

Authors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Leigh A. Wilson

Faculty of Health Science, Discipline of Behavioural and Social Sciences in Health, University of Sydney, Lidcombe, NSW, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Leigh A. Wilson .

Editor information

Editors and affiliations.

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

Wilson, L.A. (2019). Quantitative Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_54

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_54

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Fragments of peer review: A quantitative analysis of the literature (1969-2015)

Roles Conceptualization, Data curation, Formal analysis, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Department of Computer Science, University of Valencia, Burjassot, Valencian Community, Spain

Roles Formal analysis, Investigation, Methodology, Supervision, Validation, Writing – original draft

Affiliation Department of Research in Biomedicine and Health, University of Split, Split, Split-Dalmatia, Croatia

Roles Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Validation, Writing – original draft, Writing – review & editing

Affiliation Department of Economics and Management, University of Brescia, Brescia, Lombardy, Italy

• Francisco Grimaldo, 
• Ana Marušić, 
• Flaminio Squazzoni

• Published: February 21, 2018
• https://doi.org/10.1371/journal.pone.0193148
• Reader Comments

This paper examines research on peer review between 1969 and 2015 by looking at records indexed from the Scopus database. Although it is often argued that peer review has been poorly investigated, we found that the number of publications in this field doubled from 2005. A half of this work was indexed as research articles, a third as editorial notes and literature reviews and the rest were book chapters or letters. We identified the most prolific and influential scholars, the most cited publications and the most important journals in the field. Co-authorship network analysis showed that research on peer review is fragmented, with the largest group of co-authors including only 2.1% of the whole community. Co-citation network analysis indicated a fragmented structure also in terms of knowledge. This shows that despite its central role in research, peer review has been examined only through small-scale research projects. Our findings would suggest that there is need to encourage collaboration and knowledge sharing across different research communities.

Citation: Grimaldo F, Marušić A, Squazzoni F (2018) Fragments of peer review: A quantitative analysis of the literature (1969-2015). PLoS ONE 13(2): e0193148. https://doi.org/10.1371/journal.pone.0193148

Editor: Lutz Bornmann, Max Planck Society, GERMANY

Received: June 9, 2017; Accepted: February 4, 2018; Published: February 21, 2018

Copyright: © 2018 Grimaldo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This work has been supported by the TD1306 COST Action PEERE. The first author, Francisco Grimaldo, was also funded by the Spanish Ministry of Economy, Industry and Competitiveness project TIN2015-66972-C5-5-R. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Peer review is central to research. It is essential to ensure the quality of scientific publications, but also to help the scientific community self-regulate its reputation and resource allocation [ 1 ]. Whether directly or indirectly, it also influences funding and publication [ 2 ]. The transition of publishing and reading to the digital era has not changed the value of peer review, although it has stimulated the call for new models and more reliable standards [ 3 – 5 ].

Under the impact of recent scandals, where manipulated research passed the screening of peer review and was eventually published in influential journals, many analysts have suggested that more research is needed on this delicate subject [ 6 – 10 ]. The lack of data and robust evidence on the quality of the process has led many observers even to question the real value of peer review and to contemplate alternatives [ 11 – 13 ].

This study aims to provide a comprehensive analysis of peer review literature from 1969 to 2015, by looking at articles indexed in Scopus. This analysis can help to reveal the structure of the field by finding the more prolific and influential authors, the most authoritative journals and the most active research institutions. By looking at co-authorship and co-citation networks, we measured structural characteristics of the scientific community, including collaboration and knowledge sharing. This was to understand whether, despite the growing number of publications on peer review in the last years, research is excessively fragmented to give rise to a coherent and connected field.

Finally, it is important to note that the period covered by our analysis is highly representative. Indeed, although many analysts suggested that peer review is deeply rooted in the historical evolution of modern science since the 17 th century [ 14 ], recent historical analysis suggested that as an institutionalized system of evaluation in scholarly journals was established systematically only about 70 years ago, where also the terms “peer review” and “referee” became common currency [ 15 ].

Fig 1 (left panel) shows that the number of publications on peer review doubled from 2005. From 2004 to 2015, the annual growth of publications on peer review was 12% on average, reaching 28% and 38% between 2004–2005 and 2005–2006, respectively. The volume of published research grew more than the total number of publications, which had an average growth of 5% from 2004 to 2015, to reach 15% in 2004–2005 ( Fig 1 , right panel). The observed peak-based dynamics of this growth can be related to the impact of the International Congresses on Peer Review and Biomedical Publication, which were regularly held every four years starting from 1989 with the JAMA publishing abstracts and articles from the second, third and fourth editions of the congress [ 10 ]. This was also confirmed by data from PubMed and Web of Science (see Figure A in S1 Appendix ).

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

Number of records on peer review (left) and number of records published in English on any topic from 1969 to 2015 in Scopus (right).

https://doi.org/10.1371/journal.pone.0193148.g001

About half of the records were journal articles, the rest mostly being editorial notes, commentaries, letters and literature reviews (see Figure B in S1 Appendix ). However, the number of original research contributions, e.g., research articles, book chapters or conference proceedings papers, increased from 2005 onward to exceed the number of editorial notes, reviews and letters ( Fig 2 ). This would indicate that empirical, data-driven research is increasing recently.

https://doi.org/10.1371/journal.pone.0193148.g002

Fig 3 shows the top 10 most productive countries by origin of research authors. Peer review is studied predominantly in the US, followed by the UK, Australia, Canada and Germany. While this may also be due to the size-effect of these communities, previous studies have suggested that peer review is intrinsic especially to the Anglo-Saxon institutional context [ 16 ]. However, if we look at the top 10 most productive institutions, in which it is probable that research has been cumulative and more systematic, we also found two prominent European institutions, the ETH Zurich and the University of Zurich ( Fig 4 ). This indicates that research on peer review is truly international.

https://doi.org/10.1371/journal.pone.0193148.g003

https://doi.org/10.1371/journal.pone.0193148.g004

Fig 5 shows the most prolific authors. While the top ones are European social scientists, i.e., Lutz Bornmann and Hans-Dieter Daniel, who published 45 and 34 papers, respectively, the pioneers of research on peer review were medical scholars and journal editors, such as Drummond Rennie, Richard Smith and Annette Flanagin.

https://doi.org/10.1371/journal.pone.0193148.g005

Among publication outlets, the journals that published on peer review most frequently were: Science (n = 136 papers), Nature (n = 110), JAMA (n = 99), Scientometrics (n = 65), Behavioral and Brain Sciences (n = 48), Chemical and Engineering News (34), Academic Medicine (32), Australian Clinical Review (32), Learned Publishing (n = 31) and Research Evaluation (n = 31). However, research articles on peer review were published mostly by JAMA (n = 62), Behavioral and Brain Sciences (n = 44) and Scientometrics (n = 42). This means that top journals such as Science and Nature typically published commentaries, editorial notes or short surveys, while research contributions have mostly been published elsewhere. If we look at the impact of research on peer review on journal citations (see Table A in S1 Appendix ), the impact has been minimal with the exception of Scientometrics , whose articles on peer review significantly contributed to the journal’s success (10.97% of the whole journal citations were received by articles on peer review). However, it is worth noting that the contribution of research on peer review in terms of journal citations has been increasing over time (see Fig 6 for a restricted sample of journals listed in Table B in S1 Appendix ).

https://doi.org/10.1371/journal.pone.0193148.g006

Among the most important topics, looking at the keywords revealed that research has preferably examined the connection between peer review and “quality assurance” (103 papers), “publishing” (93), “research” (76), “open access” (56), “quality improvement” (47), “evaluation” (46), “publication” (44), “assessment” (41), “ethics” (40) and “bibliometrics” (39). The primacy of the link between peer review and the concept of “quality” was confirmed by looking at nouns/verbs/adjectives in the paper titles (“quality” appearing 527 times against “journal” appearing 454 times or “research”, 434 times) and in the abstracts (“quality” recurring 2208 times against “research” 2038 or medical 1014 times). This would confirm that peer review has been mainly viewed as a “quality control” process rather than a collaborative process that would aim at increasing the knowledge value of a manuscript [ 17 ].

Data showed that research on peer review is typically pursued in small collaborative networks (75% of the records had less than three co-authors), with the exception of one article published in 2012, which was co-authored by 198 authors and so was excluded by the following analysis on co-authorship networks to avoid statistical bias (see Figure D in S1 Appendix ). Around 83% of the co-authorship networks included less than six authors (see Figure E in S1 Appendix ). The most prolific authors were also those with more co-authors, although not those with a higher average number of co-authors per paper (see Table E in S1 Appendix ).

The most prolific authors from our analysis were not always those instrumental in connecting the community studying peer review (e.g., compare Table 1 and Fig 5 ). Fragmentation and small-scale collaboration networks were dominant (e.g., see Table B and Figure E in S1 Appendix ). We found 1912 clusters with an average size of 4.1, which is very small. However, it is important to emphasize certain differences in the position of scientists in these three samples. When excluding records published in medicine journals, we found a more connected co-authorship network with scientists working in more cohesive and stable groups, indicated by the lower number of clusters, higher density and shorter diameter in sample 3 in Table 2 , which is not linearly related to decreasing numbers of nodes and edges.

https://doi.org/10.1371/journal.pone.0193148.t001

https://doi.org/10.1371/journal.pone.0193148.t002

To look at this in more detail, we plot the co-authorship network linking all authors of the papers on peer review. Each network’s node was a different author and links were established between two authors whenever they co-authored a paper. The greater the number of papers co-authored between two authors, the higher the thickness of the corresponding link.

The co-authorship network was highly disaggregated, with 7971 authors forming 1910 communities ( Table 1 ). With the exception of a large community of 167 researchers and a dozen of communities including around 30 to 50 scientists, 98% of communities had fewer than 15 scientists. Note that the giant component (n = 167 scientists) represents only 2.1% of the total number of scientists in the sample. It included co-authorship relations between the top 10 most central authors and their collaborators ( Fig 7 ). The situation is different if we look at various largest communities and restrict our analysis to research articles and articles published in non-medicine journals ( Fig 8 ). In this case, collaboration groups were more cohesive (see Fig 8 , right panel).

Note that the node size refers to the author’s betweeness centrality.

https://doi.org/10.1371/journal.pone.0193148.g007

Sample 1 on the left, i.e., sample 3 on the right, i.e., outside medicine). Note that the node size refers to the author’s betweeness centrality.

https://doi.org/10.1371/journal.pone.0193148.g008

In order to look at the internal structure of the field, we built a co-citation network that measured relations between cited publications. It is important to note that here a co-citation meant that two records were cited in the same document. For the sake of clarity, we reported data only on cases in which co-citations were higher than 1.

Fig 9 shows the co-citation network, which included 6402 articles and 71548 references. In the top-right hand corner, there is the community of 84 papers, while the two small clusters at the bottom-centre and middle-left, were examples of isolated co-citation links that were generated by a small number of articles (e.g., the bottom-centre was generated by four citing articles by the same authors with a high number of co-citation links). Table 3 presents the co-citation network metrics, including data on the giant component. Results suggest that the field is characterized by network fragmentation with 192 clusters with a limited size. While the giant component covered 33% of the nodes, it counted only 0.9% of the total number of cited sources in all records. Furthermore, data showed that 79.2% of co-citation links included no more than five cited references.

https://doi.org/10.1371/journal.pone.0193148.g009

https://doi.org/10.1371/journal.pone.0193148.t003

Table 4 shows a selection of the most important references that were instrumental in clustering the co-citation network as part of the giant component. Results demonstrated not only the importance of certain classical sociology of science contributions, e.g., Robert Merton’s work, which showed an interest on peer review since the 1970s; also more recent works, including literature reviews, were important to decrease the disconnection of scientists in the field [ 2 ]. They also show that, at least for the larger co-citation subnetwork, the field is potentially inter-disciplinary, with important contributions from scholars in medicine as well as scholars from sociology and behavioural sciences.

https://doi.org/10.1371/journal.pone.0193148.t004

Discussion and conclusions

Our analysis showed that research on peer review has been rapidly growing, especially from 2005. Not only the number of publications increased; it did also the number of citations and so the impact of research on peer review [ 18 ]. We also found that research is international, with more tradition in the US but with important research groups also in Europe. However, when looking at co-authorship networks, findings indicate that research is fragmented. Scholars do not collaborate on a significant scale, with the largest group of co-authors including only 2.1% of the whole community. When looking at co-citation patterns, we found that also knowledge sharing is fragmented. The larger networks covers only 33% of the nodes, which count only for 0.9% of the total number of cited sources in all records.

This calls for a serious consideration of certain structural problems of studies on peer review. First, difficulties in accessing data from journals and funding agencies and performing large-scale quantitative research have probably limited collaborative research [ 19 ]. While the lack of data may also be due to the confidentiality and anonymity that characterize peer review, it is also possible that editorial boards of journals and administrative bodies of funding agencies have interests in obstructing independent research as a means to protect internal decisions [ 8 ]. However, the full digitalisation of editorial management processes and the increasing emphasis on open data and research integrity among science stakeholders are creating a favourable context in which researchers will be capable of accessing peer review data more frequently and easily soon [ 20 ]. This is expected to stimulate collaboration and increase the scale of research on peer review. Secondly, the lack of specific funding schemes that support research on peer review has probably obstructed the possibility of systematic studies [ 10 ]. This has probably made difficult for scholars to establish large-scale, cross-disciplinary collaboration.

In conclusion, although peer review may reflect context-specificities and disciplinary traditions, the common challenge of understanding the complexity of this process, testing the efficacy of different models in reducing bias and allocating credit and reputation fairly requires ensuring comparison and encouraging collaboration and knowledge sharing across communities [ 21 ]. Here, a recently released document on data sharing by a European project has indicated that data sharing on peer review is instrumental to promote the quality of the process, with relevant collective benefits [ 22 ]. Not only such initiatives are important to improve the quality of research; they can also promote an evidence-based approach to peer review reorganizations and innovations, which is now not so well developed.

Our sample included all records on peer review published from 1969 to 2015, which were extracted from Scopus on July 1 st 2016. We used the Advanced Search tab on the Scopus website to run our query strings (for detail, see below) and exported all available fields for each document retrieved as a CSV (comma separated values format) file. After several tests and checks on the dataset, we identified three samples of records that were hierarchically linked as follows:

• Sample 1 (n = 6402 documents), which included any paper reporting “peer review” either in the “article title” or “author keywords” fields (the use of other fields, such as “Abstract” and “Keywords”, led to a high number of documents that reported about peer review but were excluded from the sample as we verified that they were not studies on peer review but just papers that had gone through a peer review process). This sample was obtained after deduplication of the following query to Scopus: (TITLE("peer review") OR AUTHKEY("peer review")) AND PUBYEAR < 2016.
• Sample 2 (n = 3725 documents), which restricted sample 1 to journal articles (already published or just available online), books, book chapters and conference papers, so excluding editorial notes, reviews, commentaries and letters. This sample was obtained after deduplication of the following query to Scopus: (TITLE("peer review") OR AUTHKEY("peer review")) AND (DOCTYPE("ar") OR DOCTYPE ("ip") OR DOCTYPE ("bk") OR DOCTYPE ("ch") OR DOCTYPE ("cp")) AND PUBYEAR < 2016.
• Sample 3 (n = 1955 documents), which restricted sample 2 to records that were not listed among “Medicine” as subject area. This sample was obtained after deduplication of the following query to Scopus: (TITLE("peer review") OR AUTHKEY("peer review")) AND (DOCTYPE("ar") OR DOCTYPE ("ip") OR DOCTYPE ("bk") OR DOCTYPE ("ch") OR DOCTYPE ("cp")) AND PUBYEAR < 2016 AND NOT(SUBJAREA("MEDI")).

With sample 1, we aimed to exclude records that were not explicitly addressed to peer review as an object of study. With sample 2, we identified only articles that reported results, data or cases. With sample 3, we aimed to understand specificities and differences between studies on peer review in medicine and other studies. If not differently mentioned, we reported results on sample 1. Note that, in order to check data consistency, we compared our Scopus dataset with other datasets and repositories, such as PubMed and WoS (see Figure A in S1 Appendix ).

The queries to Scopus proposed in this paper allowed us to retrieve the corpus at a sufficient level of generality to look at the big picture of this field of research. Querying titles and author keywords about “peer review” did not restrict the search only to specific aspects, contexts or cases in which peer review could have been studied (e.g., peer review of scientific manuscripts). Although these queries could filter out some relevant papers, we strongly believe these cases had only a marginal impact on our analysis. For instance, we tried to use other related search terms and found a few papers from Scopus (e.g. just 2 documents for “grant decision making” and 3 documents for “grant selection”) and a number of false positives (e.g. the first 20 of the 69 documents obtained for “panel review” did not really deal with peer review as a field of research).

In order to visualize the collaboration structure in the field, we calculated co-authorship networks [ 23 ] in all samples. Each node in the co-authorship network represented a researcher, while each edge connected two researchers who co-authored a paper. In order to map knowledge sharing, we extracted co-citation networks [ 24 – 26 ]. In this case, nodes represented bibliographic references while edges connected two references when they were cited in the same paper. These methods are key to understand the emergence and evolution of research on peer review as a field driven by scientists’ connections and knowledge flows [ 27 ].

When constructing co-authorship and co-citation networks, we only used information about documents explicitly dealing with “peer review”. The rationale behind this decision was that we wanted to measure the kind of collaboration that can be attributed to these publications, regardless the total productivity of the scientists involved. Minor data inconsistencies can also happen due to the data exported from Scopus, WoS and PubMed not being complete, clean and free of errors. If a paper is missing, all co-authorship links that can be derived will be missing too. If an author name is written in two ways, two different nodes will represent the same researcher and links will be distributed between them. The continuous refinement, sophistication and precision of the algorithms behind these databases ensure that the amount of mistakes and missing information is irrelevant for a large-scale analysis. In any case, we implemented automatic mechanisms that cleaned data and removed duplicated records that reduced these inconsistencies to a marginal level given the scope of our study (see the R script used to perform the analysis in S1 File ).

Research has extensively used co-authorship and co-citation networks to study collaboration patterns by means of different network descriptors [ 28 ]. Here, we focussed on the following indicators, which were used to extract information from the samples presented above:

• Number of nodes: the number of different co-authors and different bibliographic references, which was used to provide a structural picture of the community of researchers studying peer review.
• Number of edges: the sum of all different two-by-two relationships between researchers and between bibliographic references, which was used to represent the collaboration structure of this field of research.
• Network density: the ratio between the number of edges in the co-authorship network and the total number of edges that this network would have if it were completely connected, which was used to understand whether the community was cohesive or fragmented (i.e., higher the ratio, the more cohesive was the research community).
• Diameter: the longest shortest path in the network, which indicated the distance between its two farthest nodes (i.e., two authors or two references), so showing the degree of separation in the network.
• Betweeness centrality: the number of shortest paths between any two nodes that passed through any particular node of the network. Note that nodes around the edge of the network would typically have a lower betweeness centrality, whereas a higher betweeness centrality would indicate, in our case, that a scientist or a paper was connecting, respectively, different parts of the co-authorship or the co-citation network, thus playing a central role in connecting the community.
• Number and size of clusters: here, we found the number and size of densely connected sub-communities in the network after performing a simple breadth-first search. We used these indicators to understand if the community was characterized by a multitude of sub-groups relatively independent from each other, with someone sub-connecting more researchers.

Supporting information

S1 appendix..

https://doi.org/10.1371/journal.pone.0193148.s001

S1 File. R code script used to perform the quantitative analysis.

https://doi.org/10.1371/journal.pone.0193148.s002

Acknowledgments

We would like to thank Rocio Tortajada for her help in the early stages of this research and Emilia López-Iñesta for her help in formatting the references. This paper was based upon work from the COST Action TD1306 “New Frontiers of Peer Review”-PEERE ( www.peere.org ).

• View Article
• Google Scholar
• PubMed/NCBI
• 9. The Economist. Unreliable research. Trouble at the lab. 18 Oct 2013. https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble . Cited 28 Sept 2017.
• 13. Gould THP. Do we still need peer review? An argument for change. Plymouth, UK: Scarecrow Press. 2013.
• 16. Lamont M. How professors think. Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press. 2009.
• 28. Wasserman S, Faust K. Social Network Analysis: Methods and Applications. Cambridge University Press; 1994.