example of non experimental design

6.1 Overview of Non-Experimental Research

Learning objectives.

• Define non-experimental research, distinguish it clearly from experimental research, and give several examples.
• Explain when a researcher might choose to conduct non-experimental research as opposed to experimental research.

What Is Non-Experimental Research?

Non-experimental research  is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world).

Most researchers in psychology consider the distinction between experimental and non-experimental research to be an extremely important one. This is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, non-experimental research generally cannot. As we will see, however, this inability to make causal conclusions does not mean that non-experimental research is less important than experimental research.

When to Use Non-Experimental Research

As we saw in the last chapter , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable. It stands to reason, therefore, that non-experimental research is appropriate—even necessary—when these conditions are not met. There are many times in which non-experimental research is preferred, including when:

• the research question or hypothesis relates to a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
• the research question pertains to a non-causal statistical relationship between variables (e.g., is there a correlation between verbal intelligence and mathematical intelligence?).
• the research question is about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions for practical or ethical reasons (e.g., does damage to a person’s hippocampus impair the formation of long-term memory traces?).
• the research question is broad and exploratory, or is about what it is like to have a particular experience (e.g., what is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and non-experimental approaches is generally dictated by the nature of the research question. Recall the three goals of science are to describe, to predict, and to explain. If the goal is to explain and the research question pertains to causal relationships, then the experimental approach is typically preferred. If the goal is to describe or to predict, a non-experimental approach will suffice. But the two approaches can also be used to address the same research question in complementary ways. For example, Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [1] .

Types of Non-Experimental Research

Non-experimental research falls into three broad categories: cross-sectional research, correlational research, and observational research. 

First, cross-sectional research  involves comparing two or more pre-existing groups of people. What makes this approach non-experimental is that there is no manipulation of an independent variable and no random assignment of participants to groups. Imagine, for example, that a researcher administers the Rosenberg Self-Esteem Scale to 50 American college students and 50 Japanese college students. Although this “feels” like a between-subjects experiment, it is a cross-sectional study because the researcher did not manipulate the students’ nationalities. As another example, if we wanted to compare the memory test performance of a group of cannabis users with a group of non-users, this would be considered a cross-sectional study because for ethical and practical reasons we would not be able to randomly assign participants to the cannabis user and non-user groups. Rather we would need to compare these pre-existing groups which could introduce a selection bias (the groups may differ in other ways that affect their responses on the dependent variable). For instance, cannabis users are more likely to use more alcohol and other drugs and these differences may account for differences in the dependent variable across groups, rather than cannabis use per se.

Cross-sectional designs are commonly used by developmental psychologists who study aging and by researchers interested in sex differences. Using this design, developmental psychologists compare groups of people of different ages (e.g., young adults spanning from 18-25 years of age versus older adults spanning 60-75 years of age) on various dependent variables (e.g., memory, depression, life satisfaction). Of course, the primary limitation of using this design to study the effects of aging is that differences between the groups other than age may account for differences in the dependent variable. For instance, differences between the groups may reflect the generation that people come from (a cohort effect) rather than a direct effect of age. For this reason, longitudinal studies in which one group of people is followed as they age offer a superior means of studying the effects of aging. Once again, cross-sectional designs are also commonly used to study sex differences. Since researchers cannot practically or ethically manipulate the sex of their participants they must rely on cross-sectional designs to compare groups of men and women on different outcomes (e.g., verbal ability, substance use, depression). Using these designs researchers have discovered that men are more likely than women to suffer from substance abuse problems while women are more likely than men to suffer from depression. But, using this design it is unclear what is causing these differences. So, using this design it is unclear whether these differences are due to environmental factors like socialization or biological factors like hormones?

When researchers use a participant characteristic to create groups (nationality, cannabis use, age, sex), the independent variable is usually referred to as an experimenter-selected independent variable (as opposed to the experimenter-manipulated independent variables used in experimental research). Figure 6.1 shows data from a hypothetical study on the relationship between whether people make a daily list of things to do (a “to-do list”) and stress. Notice that it is unclear whether this is an experiment or a cross-sectional study because it is unclear whether the independent variable was manipulated by the researcher or simply selected by the researcher. If the researcher randomly assigned some participants to make daily to-do lists and others not to, then the independent variable was experimenter-manipulated and it is a true experiment. If the researcher simply asked participants whether they made daily to-do lists or not, then the independent variable it is experimenter-selected and the study is cross-sectional. The distinction is important because if the study was an experiment, then it could be concluded that making the daily to-do lists reduced participants’ stress. But if it was a cross-sectional study, it could only be concluded that these variables are statistically related. Perhaps being stressed has a negative effect on people’s ability to plan ahead. Or perhaps people who are more conscientious are more likely to make to-do lists and less likely to be stressed. The crucial point is that what defines a study as experimental or cross-sectional l is not the variables being studied, nor whether the variables are quantitative or categorical, nor the type of graph or statistics used to analyze the data. It is how the study is conducted.

Figure 6.1  Results of a Hypothetical Study on Whether People Who Make Daily To-Do Lists Experience Less Stress Than People Who Do Not Make Such Lists

Second, the most common type of non-experimental research conducted in Psychology is correlational research. Correlational research is considered non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable.  More specifically, in correlational research , the researcher measures two continuous variables with little or no attempt to control extraneous variables and then assesses the relationship between them. As an example, a researcher interested in the relationship between self-esteem and school achievement could collect data on students’ self-esteem and their GPAs to see if the two variables are statistically related. Correlational research is very similar to cross-sectional research, and sometimes these terms are used interchangeably. The distinction that will be made in this book is that, rather than comparing two or more pre-existing groups of people as is done with cross-sectional research, correlational research involves correlating two continuous variables (groups are not formed and compared).

Third,   observational research  is non-experimental because it focuses on making observations of behavior in a natural or laboratory setting without manipulating anything. Milgram’s original obedience study was non-experimental in this way. He was primarily interested in the extent to which participants obeyed the researcher when he told them to shock the confederate and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of observational research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the researchers asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.

The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. But as you will learn in this chapter, many observational research studies are more qualitative in nature. In  qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s observational study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semi-public room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256) [2] . Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable.  Figure 6.2  shows how experimental, quasi-experimental, and non-experimental (correlational) research vary in terms of internal validity. Experimental research tends to be highest in internal validity because the use of manipulation (of the independent variable) and control (of extraneous variables) help to rule out alternative explanations for the observed relationships. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Non-experimental (correlational) research is lowest in internal validity because these designs fail to use manipulation or control. Quasi-experimental research (which will be described in more detail in a subsequent chapter) is in the middle because it contains some, but not all, of the features of a true experiment. For instance, it may fail to use random assignment to assign participants to groups or fail to use counterbalancing to control for potential order effects. Imagine, for example, that a researcher finds two similar schools, starts an anti-bullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” While a comparison is being made with a control condition, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying (e.g., there may be a selection effect).

Figure 6.2 Internal Validity of Correlation, Quasi-Experimental, and Experimental Studies. Experiments are generally high in internal validity, quasi-experiments lower, and correlation studies lower still.

Notice also in  Figure 6.2  that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well-designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

Key Takeaways

• Non-experimental research is research that lacks the manipulation of an independent variable.
• There are two broad types of non-experimental research. Correlational research that focuses on statistical relationships between variables that are measured but not manipulated, and observational research in which participants are observed and their behavior is recorded without the researcher interfering or manipulating any variables.
• In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.
• A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
• A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
• A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
• A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.
• Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

Share This Book

• Increase Font Size

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.1 Overview of Nonexperimental Research

Learning objectives.

• Define nonexperimental research, distinguish it clearly from experimental research, and give several examples.
• Explain when a researcher might choose to conduct nonexperimental research as opposed to experimental research.

What Is Nonexperimental Research?

Nonexperimental research is research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

In a sense, it is unfair to define this large and diverse set of approaches collectively by what they are not . But doing so reflects the fact that most researchers in psychology consider the distinction between experimental and nonexperimental research to be an extremely important one. This is because while experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, nonexperimental research generally cannot. As we will see, however, this does not mean that nonexperimental research is less important than experimental research or inferior to it in any general sense.

When to Use Nonexperimental Research

As we saw in Chapter 6 “Experimental Research” , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable and randomly assign participants to conditions or to orders of conditions. It stands to reason, therefore, that nonexperimental research is appropriate—even necessary—when these conditions are not met. There are many ways in which this can be the case.

• The research question or hypothesis can be about a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
• The research question can be about a noncausal statistical relationship between variables (e.g., Is there a correlation between verbal intelligence and mathematical intelligence?).
• The research question can be about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions (e.g., Does damage to a person’s hippocampus impair the formation of long-term memory traces?).
• The research question can be broad and exploratory, or it can be about what it is like to have a particular experience (e.g., What is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and nonexperimental approaches is generally dictated by the nature of the research question. If it is about a causal relationship and involves an independent variable that can be manipulated, the experimental approach is typically preferred. Otherwise, the nonexperimental approach is preferred. But the two approaches can also be used to address the same research question in complementary ways. For example, nonexperimental studies establishing that there is a relationship between watching violent television and aggressive behavior have been complemented by experimental studies confirming that the relationship is a causal one (Bushman & Huesmann, 2001). Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974).

Types of Nonexperimental Research

Nonexperimental research falls into three broad categories: single-variable research, correlational and quasi-experimental research, and qualitative research. First, research can be nonexperimental because it focuses on a single variable rather than a statistical relationship between two variables. Although there is no widely shared term for this kind of research, we will call it single-variable research . Milgram’s original obedience study was nonexperimental in this way. He was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of single-variable research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the research asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.)

As these examples make clear, single-variable research can answer interesting and important questions. What it cannot do, however, is answer questions about statistical relationships between variables. This is a point that beginning researchers sometimes miss. Imagine, for example, a group of research methods students interested in the relationship between children’s being the victim of bullying and the children’s self-esteem. The first thing that is likely to occur to these researchers is to obtain a sample of middle-school students who have been bullied and then to measure their self-esteem. But this would be a single-variable study with self-esteem as the only variable. Although it would tell the researchers something about the self-esteem of children who have been bullied, it would not tell them what they really want to know, which is how the self-esteem of children who have been bullied compares with the self-esteem of children who have not. Is it lower? Is it the same? Could it even be higher? To answer this question, their sample would also have to include middle-school students who have not been bullied.

Research can also be nonexperimental because it focuses on a statistical relationship between two variables but does not include the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both. This kind of research takes two basic forms: correlational research and quasi-experimental research. In correlational research , the researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them. A research methods student who finds out whether each of several middle-school students has been bullied and then measures each student’s self-esteem is conducting correlational research. In quasi-experimental research , the researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions. For example, a researcher might start an antibullying program (a kind of treatment) at one school and compare the incidence of bullying at that school with the incidence at a similar school that has no antibullying program.

The final way in which research can be nonexperimental is that it can be qualitative. The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. In qualitative research , the data are usually nonnumerical and are analyzed using nonstatistical techniques. Rosenhan’s study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semipublic room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256).

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable. Figure 7.1 shows how experimental, quasi-experimental, and correlational research vary in terms of internal validity. Experimental research tends to be highest because it addresses the directionality and third-variable problems through manipulation and the control of extraneous variables through random assignment. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Correlational research is lowest because it fails to address either problem. If the average score on the dependent variable differs across levels of the independent variable, it could be that the independent variable is responsible, but there are other interpretations. In some situations, the direction of causality could be reversed. In others, there could be a third variable that is causing differences in both the independent and dependent variables. Quasi-experimental research is in the middle because the manipulation of the independent variable addresses some problems, but the lack of random assignment and experimental control fails to address others. Imagine, for example, that a researcher finds two similar schools, starts an antibullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” There is no directionality problem because clearly the number of bullying incidents did not determine which school got the program. However, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying.

Experiments are generally high in internal validity, quasi-experiments lower, and correlational studies lower still.

Notice also in Figure 7.1 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well designed quasi-experiment with no obvious confounding variables.

Key Takeaways

• Nonexperimental research is research that lacks the manipulation of an independent variable, control of extraneous variables through random assignment, or both.
• There are three broad types of nonexperimental research. Single-variable research focuses on a single variable rather than a relationship between variables. Correlational and quasi-experimental research focus on a statistical relationship but lack manipulation or random assignment. Qualitative research focuses on broader research questions, typically involves collecting large amounts of data from a small number of participants, and analyzes the data nonstatistically.
• In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.

Discussion: For each of the following studies, decide which type of research design it is and explain why.

• A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
• A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
• A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
• A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.

Bushman, B. J., & Huesmann, L. R. (2001). Effects of televised violence on aggression. In D. Singer & J. Singer (Eds.), Handbook of children and the media (pp. 223–254). Thousand Oaks, CA: Sage.

Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row.

Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Yearly plans are up to 65% off for a limited Black Friday sale. ⏰

• Form Builder
• Survey Maker
• AI Form Generator
• AI Survey Tool
• AI Quiz Maker
• Store Builder
• WordPress Plugin

HubSpot CRM

Google Sheets

Google Analytics

Microsoft Excel

• Popular Forms
• Job Application Form Template
• Rental Application Form Template
• Hotel Accommodation Form Template
• Online Registration Form Template
• Employment Application Form Template
• Application Forms
• Booking Forms
• Consent Forms
• Contact Forms
• Donation Forms
• Customer Satisfaction Surveys
• Employee Satisfaction Surveys
• Evaluation Surveys
• Feedback Surveys
• Market Research Surveys
• Personality Quiz Template
• Geography Quiz Template
• Math Quiz Template
• Science Quiz Template
• Vocabulary Quiz Template

Try without registration Quick Start

Read engaging stories, how-to guides, learn about forms.app features.

Inspirational ready-to-use templates for getting started fast and powerful.

Spot-on guides on how to use forms.app and make the most out of it.

See the technical measures we take and learn how we keep your data safe and secure.

• Integrations
• Help Center
• Sign In Sign Up Free
• What is non-experimental research: Definition, types & examples

Defne Çobanoğlu

The experimentation method is very useful for getting information on a specific subject. However, when experimenting is not possible or practical, there is another way of collecting data for those interested. It's a non-experimental way, to say the least.

In this article, we have gathered information on non-experimental research, clearly defined what it is and when one should use it, and listed the types of non-experimental research. We also gave some useful examples to paint a better picture. Let us get started. 

• What is non-experimental research?

Non-experimental research is a type of research design that is based on observation and measuring instead of experimentation with randomly assigned participants.

What characterizes this research design is the fact that it lacks the manipulation of independent variables . Because of this fact, the non-experimental research is based on naturally occurring conditions, and there is no involvement of external interventions. Therefore, the researchers doing this method must not rely heavily on interviews, surveys , or case studies.

• When to use non-experimental research?

An experiment is done when a researcher is investigating the relationship between one or two phenomena and has a theory or hypothesis on the relationship between two variables that are involved. The researcher can carry out an experiment when it is ethical, possible, and feasible to do one.

However, when an experiment can not be done because of a limitation, then they decide to opt for a non-experimental research design . Non-experimental research is considered preferable in some conditions, including:

• When the manipulation of the independent variable is not possible because of ethical or practical concerns
• When the subjects of an experimental design can not be randomly assigned to treatments.
• When the research question is too extensive or it relates to a general experience.
• When researchers want to do a starter research before investing in more extensive research.
• When the research question is about the statistical relationship between variables , but in a noncausal context.
• Characteristics of non-experimental research

Non-experimental research has some characteristics that clearly define the framework of this research method. They provide a clear distinction between experimental design and non-experimental design. Let us see some of them:

• Non-experimental research does not involve the manipulation of variables .
• The aim of this research type is to explore the factors as they naturally occur .
• This method is used when experimentation is not possible because of ethical or practical reasons .
• Instead of creating a sample or participant group, the existing groups or natural thresholds are used during the research.
• This research method is not about finding causality between two variables.
• Most studies are done on past events or historical occurrences to make sense of specific research questions.
• Types of non-experimental research

Non-experimental research types

What makes research non-experimental research is the fact that the researcher does not manipulate the factors, does not randomly assign the participants, and observes the existing groups. But this research method can also be divided into different types. These types are:

Correlational research:

In correlation studies, the researcher does not manipulate the variables and is not interested in controlling the extraneous variables. They only observe and assess the relationship between them. For example, a researcher examines students’ study hours every day and their overall academic performance. The positive correlation this between study hours and academic performance suggests a statistical association. 

Quasi-experimental research:

In quasi-experimental research, the researcher does not randomly assign the participants into two groups. Because you can not deliberately deprive someone of treatment, the researcher uses natural thresholds or dividing points . For example, examining students from two different high schools with different education methods.

Cross-sectional research:

In cross-sectional research, the researcher studies and compares a portion of a population at the same time . It does not involve random assignment or any outside manipulation. For example, a study on smokers and non-smokers in a specific area.

Observational research:

In observational research, the researcher once again does not manipulate any aspect of the study, and their main focus is observation of the participants . For example, a researcher examining a group of children playing in a playground would be a good example.

• Non-experimental research examples

Non-experimental research is a good way of collecting information and exploring relationships between variables. It can be used in numerous fields, from social sciences, economics, psychology, education, and market research. When gathering information using secondary research is not enough and an experiment can not be done, this method can bring out new information.

Non-experimental research example #1

Imagine a researcher who wants to see the connection between mobile phone usage before bedtime and the amount of sleep adults get in a night . They can gather a group of individuals to observe and present them with some questions asking about the details of their day, frequency and duration of phone usage, quality of sleep, etc . And observe them by analyzing the findings.

Non-experimental research example #2

Imagine a researcher who wants to explore the correlation between job satisfaction levels among employees and what are the factors that affect this . The researcher can gather all the information they get about the employees’ ages, sexes, positions in the company, working patterns, demographic information, etc . 

The research provides the researcher with all the information to make an analysis to identify correlations and patterns. Then, it is possible for researchers and administrators to make informed predictions.

• Frequently asked questions about non-experimental research

When not to use non-experimental research?

There are some situations where non-experimental research is not suitable or the best choice. For example, the aim of non-experimental research is not about finding causality therefore, if the researcher wants to explore the relationship between two variables, then this method is not for them. Also, if the control over the variables is extremely important to the test of a theory, then experimentation is a more appropriate option.

What is the difference between experimental and non-experimental research?

Experimental research is an example of primary research where the researcher takes control of all the variables, randomly assigns the participants into different groups, and studies them in a pre-determined environment to test a hypothesis. 

On the contrary, non-experimental research does not intervene in any way and only observes and studies the participants in their natural environments to make sense of a phenomenon

What makes a quasi-experiment a non-experiment?

The same as true experimentation, quasi-experiment research also aims to explore a cause-and-effect relationship between independent and dependent variables. However, in quasi-experimental research, the participants are not randomly selected. They are assigned to groups based on non-random criteria .

Is a survey a non-experimental study?

Yes, as the main purpose of a survey or questionnaire is to collect information from participants without outside interference, it makes the survey a non-experimental study. Surveys are used by researchers when experimentation is not possible because of ethical reasons, but first-hand data is needed

What is non-experimental data?

Non-experimental data is data collected by researchers via using non-experimental methods such as observations, interpretation, and interactions. Non-experimental data could both be qualitative or quantitative, depending on the situation.

Advantages of non-experimental research

Non-experimental research has its positive sides that a researcher should have in mind when going through a study. They can start their research by going through the advantages. These advantages are:

• It is used to observe and analyze past events .
• This method is more affordable than a true experiment .
• As the researcher can adapt the methods during the study, this research type is more flexible than an experimental study.
• This method allows the researchers to answer specific questions .

Disadvantages of non-experimental research

Even though non-experimental research has its advantages, it also has some disadvantages a researcher should be mindful of. Here are some of them:

• The findings of non-experimental research can not be generalized to the whole population. Therefore, it has low external validity .
• This research is used to explore only a single variable .
• Non-experimental research designs are prone to researcher bias and may not produce neutral results.
• Final words

A non-experimental study differs from an experimental study in that there is no intervention or change of internal or extraneous elements. It is a smart way to collect information without the limitations of experimentation. These limitations could be about ethical or practical problems. When you can not do proper experimentation, your other option is to study existing conditions and groups to draw conclusions. This is a non-experimental design .

In this article, we have gathered information on non-experimental research to shed light on the details of this research method. If you are thinking of doing a study, make sure to have this information in mind. And lastly, do not forget to visit our articles on other research methods and so much more!

Defne is a content writer at forms.app. She is also a translator specializing in literary translation. Defne loves reading, writing, and translating professionally and as a hobby. Her expertise lies in survey research, research methodologies, content writing, and translation.

• Form Features
• Data Collection

Table of Contents

Related posts.

110+ Movie quiz questions to ask (+Free templates & more)

35+ Hilarious agency memes that are so true it kind of hurts

Şeyma Beyazçiçek

7 Best fyrebox alternatives & competitiors to create quizzes

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 7: Nonexperimental Research

Overview of Nonexperimental Research

Learning Objectives

• Define nonexperimental research, distinguish it clearly from experimental research, and give several examples.
• Explain when a researcher might choose to conduct nonexperimental research as opposed to experimental research.

What Is Nonexperimental Research?

Nonexperimental research  is research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

In a sense, it is unfair to define this large and diverse set of approaches collectively by what they are  not . But doing so reflects the fact that most researchers in psychology consider the distinction between experimental and nonexperimental research to be an extremely important one. This distinction is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, nonexperimental research generally cannot. As we will see, however, this inability does not mean that nonexperimental research is less important than experimental research or inferior to it in any general sense.

When to Use Nonexperimental Research

As we saw in  Chapter 6 , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable and randomly assign participants to conditions or to orders of conditions. It stands to reason, therefore, that nonexperimental research is appropriate—even necessary—when these conditions are not met. There are many ways in which preferring nonexperimental research can be the case.

• The research question or hypothesis can be about a single variable rather than a statistical relationship between two variables (e.g., How accurate are people’s first impressions?).
• The research question can be about a noncausal statistical relationship between variables (e.g., Is there a correlation between verbal intelligence and mathematical intelligence?).
• The research question can be about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions (e.g., Does damage to a person’s hippocampus impair the formation of long-term memory traces?).
• The research question can be broad and exploratory, or it can be about what it is like to have a particular experience (e.g., What is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and nonexperimental approaches is generally dictated by the nature of the research question. If it is about a causal relationship and involves an independent variable that can be manipulated, the experimental approach is typically preferred. Otherwise, the nonexperimental approach is preferred. But the two approaches can also be used to address the same research question in complementary ways. For example, nonexperimental studies establishing that there is a relationship between watching violent television and aggressive behaviour have been complemented by experimental studies confirming that the relationship is a causal one (Bushman & Huesmann, 2001) [1] . Similarly, after his original study, Milgram conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [2] .

Types of Nonexperimental Research

Nonexperimental research falls into three broad categories: single-variable research, correlational and quasi-experimental research, and qualitative research. First, research can be nonexperimental because it focuses on a single variable rather than a statistical relationship between two variables. Although there is no widely shared term for this kind of research, we will call it  single-variable research . Milgram’s original obedience study was nonexperimental in this way. He was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of single-variable research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the research asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories.)

As these examples make clear, single-variable research can answer interesting and important questions. What it cannot do, however, is answer questions about statistical relationships between variables. This detail is a point that beginning researchers sometimes miss. Imagine, for example, a group of research methods students interested in the relationship between children’s being the victim of bullying and the children’s self-esteem. The first thing that is likely to occur to these researchers is to obtain a sample of middle-school students who have been bullied and then to measure their self-esteem. But this design would be a single-variable study with self-esteem as the only variable. Although it would tell the researchers something about the self-esteem of children who have been bullied, it would not tell them what they really want to know, which is how the self-esteem of children who have been bullied  compares  with the self-esteem of children who have not. Is it lower? Is it the same? Could it even be higher? To answer this question, their sample would also have to include middle-school students who have not been bullied thereby introducing another variable.

Research can also be nonexperimental because it focuses on a statistical relationship between two variables but does not include the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both. This kind of research takes two basic forms: correlational research and quasi-experimental research. In correlational research , the researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them. A research methods student who finds out whether each of several middle-school students has been bullied and then measures each student’s self-esteem is conducting correlational research. In  quasi-experimental research , the researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions. For example, a researcher might start an antibullying program (a kind of treatment) at one school and compare the incidence of bullying at that school with the incidence at a similar school that has no antibullying program.

The final way in which research can be nonexperimental is that it can be qualitative. The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. In  qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s study of the experience of people in a psychiatric ward was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semipublic room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256). [3] Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable.  Figure 7.1  shows how experimental, quasi-experimental, and correlational research vary in terms of internal validity. Experimental research tends to be highest because it addresses the directionality and third-variable problems through manipulation and the control of extraneous variables through random assignment. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Correlational research is lowest because it fails to address either problem. If the average score on the dependent variable differs across levels of the independent variable, it  could  be that the independent variable is responsible, but there are other interpretations. In some situations, the direction of causality could be reversed. In others, there could be a third variable that is causing differences in both the independent and dependent variables. Quasi-experimental research is in the middle because the manipulation of the independent variable addresses some problems, but the lack of random assignment and experimental control fails to address others. Imagine, for example, that a researcher finds two similar schools, starts an antibullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” There is no directionality problem because clearly the number of bullying incidents did not determine which school got the program. However, the lack of random assignment of children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying.

Notice also in  Figure 7.1  that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in  Chapter 5.

Key Takeaways

• Nonexperimental research is research that lacks the manipulation of an independent variable, control of extraneous variables through random assignment, or both.
• There are three broad types of nonexperimental research. Single-variable research focuses on a single variable rather than a relationship between variables. Correlational and quasi-experimental research focus on a statistical relationship but lack manipulation or random assignment. Qualitative research focuses on broader research questions, typically involves collecting large amounts of data from a small number of participants, and analyses the data nonstatistically.
• In general, experimental research is high in internal validity, correlational research is low in internal validity, and quasi-experimental research is in between.

Discussion: For each of the following studies, decide which type of research design it is and explain why.

• A researcher conducts detailed interviews with unmarried teenage fathers to learn about how they feel and what they think about their role as fathers and summarizes their feelings in a written narrative.
• A researcher measures the impulsivity of a large sample of drivers and looks at the statistical relationship between this variable and the number of traffic tickets the drivers have received.
• A researcher randomly assigns patients with low back pain either to a treatment involving hypnosis or to a treatment involving exercise. She then measures their level of low back pain after 3 months.
• A college instructor gives weekly quizzes to students in one section of his course but no weekly quizzes to students in another section to see whether this has an effect on their test performance.
• Bushman, B. J., & Huesmann, L. R. (2001). Effects of televised violence on aggression. In D. Singer & J. Singer (Eds.), Handbook of children and the media (pp. 223–254). Thousand Oaks, CA: Sage. ↵
• Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

Research that lacks the manipulation of an independent variable, random assignment of participants to conditions or orders of conditions, or both.

Research that focuses on a single variable rather than a statistical relationship between two variables.

The researcher measures the two variables of interest with little or no attempt to control extraneous variables and then assesses the relationship between them.

The researcher manipulates an independent variable but does not randomly assign participants to conditions or orders of conditions.

Research Methods in Psychology - 2nd Canadian Edition Copyright © 2015 by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Experimental Vs Non-Experimental Research: 15 Key Differences

There is a general misconception around research that once the research is non-experimental, then it is non-scientific, making it more important to understand what experimental and experimental research entails. Experimental research is the most common type of research, which a lot of people refer to as scientific research. 

Non experimental research, on the other hand, is easily used to classify research that is not experimental. It clearly differs from experimental research, and as such has different use cases. 

In this article, we will be explaining these differences in detail so as to ensure proper identification during the research process.

What is Experimental Research?  

Experimental research is the type of research that uses a scientific approach towards manipulating one or more control variables of the research subject(s) and measuring the effect of this manipulation on the subject. It is known for the fact that it allows the manipulation of control variables. 

This research method is widely used in various physical and social science fields, even though it may be quite difficult to execute. Within the information field, they are much more common in information systems research than in library and information management research.

Experimental research is usually undertaken when the goal of the research is to trace cause-and-effect relationships between defined variables. However, the type of experimental research chosen has a significant influence on the results of the experiment.

Therefore bringing us to the different types of experimental research. There are 3 main types of experimental research, namely; pre-experimental, quasi-experimental, and true experimental research.

Pre-experimental Research

Pre-experimental research is the simplest form of research, and is carried out by observing a group or groups of dependent variables after the treatment of an independent variable which is presumed to cause change on the group(s). It is further divided into three types.

• One-shot case study research 
• One-group pretest-posttest research 
• Static-group comparison

Quasi-experimental Research

The Quasi type of experimental research is similar to true experimental research, but uses carefully selected rather than randomized subjects. The following are examples of quasi-experimental research:

• Time series 
• No equivalent control group design
• Counterbalanced design.

True Experimental Research

True experimental research is the most accurate type,  and may simply be called experimental research. It manipulates a control group towards a group of randomly selected subjects and records the effect of this manipulation.

True experimental research can be further classified into the following groups:

• The posttest-only control group 
• The pretest-posttest control group 
• Solomon four-group 

Pros of True Experimental Research

• Researchers can have control over variables.
• It can be combined with other research methods.
• The research process is usually well structured.
• It provides specific conclusions.
• The results of experimental research can be easily duplicated.

Cons of True Experimental Research

• It is highly prone to human error.
• Exerting control over extraneous variables may lead to the personal bias of the researcher.
• It is time-consuming.
• It is expensive. 
• Manipulating control variables may have ethical implications.
• It produces artificial results.

What is Non-Experimental Research?  

Non-experimental research is the type of research that does not involve the manipulation of control or independent variable. In non-experimental research, researchers measure variables as they naturally occur without any further manipulation.

This type of research is used when the researcher has no specific research question about a causal relationship between 2 different variables, and manipulation of the independent variable is impossible. They are also used when:

• subjects cannot be randomly assigned to conditions.
• the research subject is about a causal relationship but the independent variable cannot be manipulated.
• the research is broad and exploratory
• the research pertains to a non-causal relationship between variables.
• limited information can be accessed about the research subject.

There are 3 main types of non-experimental research , namely; cross-sectional research, correlation research, and observational research.

Cross-sectional Research

Cross-sectional research involves the comparison of two or more pre-existing groups of people under the same criteria. This approach is classified as non-experimental because the groups are not randomly selected and the independent variable is not manipulated.

For example, an academic institution may want to reward its first-class students with a scholarship for their academic excellence. Therefore, each faculty places students in the eligible and ineligible group according to their class of degree.

In this case, the student’s class of degree cannot be manipulated to qualify him or her for a scholarship because it is an unethical thing to do. Therefore, the placement is cross-sectional.

Correlational Research

Correlational type of research compares the statistical relationship between two variables .Correlational research is classified as non-experimental because it does not manipulate the independent variables.

For example, a researcher may wish to investigate the relationship between the class of family students come from and their grades in school. A questionnaire may be given to students to know the average income of their family, then compare it with CGPAs. 

The researcher will discover whether these two factors are positively correlated, negatively corrected, or have zero correlation at the end of the research.

Observational Research

Observational research focuses on observing the behavior of a research subject in a natural or laboratory setting. It is classified as non-experimental because it does not involve the manipulation of independent variables.

A good example of observational research is an investigation of the crowd effect or psychology in a particular group of people. Imagine a situation where there are 2 ATMs at a place, and only one of the ATMs is filled with a queue, while the other is abandoned.

The crowd effect infers that the majority of newcomers will also abandon the other ATM.

You will notice that each of these non-experimental research is descriptive in nature. It then suffices to say that descriptive research is an example of non-experimental research.

Pros of Observational Research

• The research process is very close to a real-life situation.
• It does not allow for the manipulation of variables due to ethical reasons.
• Human characteristics are not subject to experimental manipulation.

Cons of Observational Research

• The groups may be dissimilar and nonhomogeneous because they are not randomly selected, affecting the authenticity and generalizability of the study results.
• The results obtained cannot be absolutely clear and error-free.

What Are The Differences Between Experimental and Non-Experimental Research?    

• Definitions

Experimental research is the type of research that uses a scientific approach towards manipulating one or more control variables and measuring their defect on the dependent variables, while non-experimental research is the type of research that does not involve the manipulation of control variables.

The main distinction in these 2 types of research is their attitude towards the manipulation of control variables. Experimental allows for the manipulation of control variables while non-experimental research doesn’t.

 Examples of experimental research are laboratory experiments that involve mixing different chemical elements together to see the effect of one element on the other while non-experimental research examples are investigations into the characteristics of different chemical elements.

Consider a researcher carrying out a laboratory test to determine the effect of adding Nitrogen gas to Hydrogen gas. It may be discovered that using the Haber process, one can create Nitrogen gas.

Non-experimental research may further be carried out on Ammonia, to determine its characteristics, behaviour, and nature.

There are 3 types of experimental research, namely; experimental research, quasi-experimental research, and true experimental research. Although also 3 in number, non-experimental research can be classified into cross-sectional research, correlational research, and observational research.

The different types of experimental research are further divided into different parts, while non-experimental research types are not further divided. Clearly, these divisions are not the same in experimental and non-experimental research.

• Characteristics

Experimental research is usually quantitative, controlled, and multivariable. Non-experimental research can be both quantitative and qualitative , has an uncontrolled variable, and also a cross-sectional research problem.

The characteristics of experimental research are the direct opposite of that of non-experimental research. The most distinct characteristic element is the ability to control or manipulate independent variables in experimental research and not in non-experimental research. 

In experimental research, a level of control is usually exerted on extraneous variables, therefore tampering with the natural research setting. Experimental research settings are usually more natural with no tampering with the extraneous variables.

• Data Collection/Tools

  The data used during experimental research is collected through observational study, simulations, and surveys while non-experimental data is collected through observations, surveys, and case studies. The main distinction between these data collection tools is case studies and simulations.

Even at that, similar tools are used differently. For example, an observational study may be used during a laboratory experiment that tests how the effect of a control variable manifests over a period of time in experimental research. 

However, when used in non-experimental research, data is collected based on the researcher’s discretion and not through a clear scientific reaction. In this case, we see a difference in the level of objectivity. 

The goal of experimental research is to measure the causes and effects of variables present in research, while non-experimental research provides very little to no information about causal agents.

Experimental research answers the question of why something is happening. This is quite different in non-experimental research, as they are more descriptive in nature with the end goal being to describe what .

 Experimental research is mostly used to make scientific innovations and find major solutions to problems while non-experimental research is used to define subject characteristics, measure data trends, compare situations and validate existing conditions.

For example, if experimental research results in an innovative discovery or solution, non-experimental research will be conducted to validate this discovery. This research is done for a period of time in order to properly study the subject of research.

Experimental research process is usually well structured and as such produces results with very little to no errors, while non-experimental research helps to create real-life related experiments. There are a lot more advantages of experimental and non-experimental research , with the absence of each of these advantages in the other leaving it at a disadvantage.

For example, the lack of a random selection process in non-experimental research leads to the inability to arrive at a generalizable result. Similarly, the ability to manipulate control variables in experimental research may lead to the personal bias of the researcher.

• Disadvantage

 Experimental research is highly prone to human error while the major disadvantage of non-experimental research is that the results obtained cannot be absolutely clear and error-free. In the long run, the error obtained due to human error may affect the results of the experimental research.

Some other disadvantages of experimental research include the following; extraneous variables cannot always be controlled, human responses can be difficult to measure, and participants may also cause bias.

  In experimental research, researchers can control and manipulate control variables, while in non-experimental research, researchers cannot manipulate these variables. This cannot be done due to ethical reasons. 

For example, when promoting employees due to how well they did in their annual performance review, it will be unethical to manipulate the results of the performance review (independent variable). That way, we can get impartial results of those who deserve a promotion and those who don’t.

Experimental researchers may also decide to eliminate extraneous variables so as to have enough control over the research process. Once again, this is something that cannot be done in non-experimental research because it relates more to real-life situations.

Experimental research is carried out in an unnatural setting because most of the factors that influence the setting are controlled while the non-experimental research setting remains natural and uncontrolled. One of the things usually tampered with during research is extraneous variables.

In a bid to get a perfect and well-structured research process and results, researchers sometimes eliminate extraneous variables. Although sometimes seen as insignificant, the elimination of these variables may affect the research results.

Consider the optimization problem whose aim is to minimize the cost of production of a car, with the constraints being the number of workers and the number of hours they spend working per day. 

In this problem, extraneous variables like machine failure rates or accidents are eliminated. In the long run, these things may occur and may invalidate the result.

• Cause-Effect Relationship

The relationship between cause and effect is established in experimental research while it cannot be established in non-experimental research. Rather than establish a cause-effect relationship, non-experimental research focuses on providing descriptive results.

Although it acknowledges the causal variable and its effect on the dependent variables, it does not measure how or the extent to which these dependent variables change. It, however, observes these changes, compares the changes in 2 variables, and describes them.

Experimental research does not compare variables while non-experimental research does. It compares 2 variables and describes the relationship between them.

The relationship between these variables can be positively correlated, negatively correlated or not correlated at all. For example, consider a case whereby the subject of research is a drum, and the control or independent variable is the drumstick.

Experimental research will measure the effect of hitting the drumstick on the drum, where the result of this research will be sound. That is, when you hit a drumstick on a drum, it makes a sound.

Non-experimental research, on the other hand, will investigate the correlation between how hard the drum is hit and the loudness of the sound that comes out. That is, if the sound will be higher with a harder bang, lower with a harder bang, or will remain the same no matter how hard we hit the drum.

• Quantitativeness

Experimental research is a quantitative research method while non-experimental research can be both quantitative and qualitative depending on the time and the situation where it is been used. An example of a non-experimental quantitative research method is correlational research .

Researchers use it to correlate two or more variables using mathematical analysis methods. The original patterns, relationships, and trends between variables are observed, then the impact of one of these variables on the other is recorded along with how it changes the relationship between the two variables.

Observational research is an example of non-experimental research, which is classified as a qualitative research method.

• Cross-section

Experimental research is usually single-sectional while non-experimental research is cross-sectional. That is, when evaluating the research subjects in experimental research, each group is evaluated as an entity.

For example, let us consider a medical research process investigating the prevalence of breast cancer in a certain community. In this community, we will find people of different ages, ethnicities, and social backgrounds. 

If a significant amount of women from a particular age are found to be more prone to have the disease, the researcher can conduct further studies to understand the reason behind it. A further study into this will be experimental and the subject won’t be a cross-sectional group. 

A lot of researchers consider the distinction between experimental and non-experimental research to be an extremely important one. This is partly due to the fact that experimental research can accommodate the manipulation of independent variables, which is something non-experimental research can not.

Therefore, as a researcher who is interested in using any one of experimental and non-experimental research, it is important to understand the distinction between these two. This helps in deciding which method is better for carrying out particular research. 

Connect to Formplus, Get Started Now - It's Free!

• examples of experimental research
• non experimental research
• busayo.longe

You may also like:

Response vs Explanatory Variables: Definition & Examples

In this article, we’ll be comparing the two types of variables, what they both mean and see some of their real-life applications in research

Experimental Research Designs: Types, Examples & Methods

Ultimate guide to experimental research. It’s definition, types, characteristics, uses, examples and methodolgy

What is Experimenter Bias? Definition, Types & Mitigation

In this article, we will look into the concept of experimental bias and how it can be identified in your research

Simpson’s Paradox & How to Avoid it in Experimental Research

In this article, we are going to look at Simpson’s Paradox from its historical point and later, we’ll consider its effect in...

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Non-experimental research: What it is, overview & advantages

Non-experimental research is the type of research that lacks an independent variable. Instead, the researcher observes the context in which the phenomenon occurs and analyzes it to obtain information.

Unlike experimental research , where the variables are held constant, non-experimental research happens during the study when the researcher cannot control, manipulate or alter the subjects but relies on interpretation or observations to conclude.

This means that the method must not rely on correlations, surveys , or case studies and cannot demonstrate an actual cause and effect relationship.

Characteristics of non-experimental research

Some of the essential characteristics of non-experimental research are necessary for the final results. Let’s talk about them to identify the most critical parts of them.

• Most studies are based on events that occurred previously and are analyzed later.
• In this method, controlled experiments are not performed for reasons such as ethics or morality.
• No study samples are created; on the contrary, the samples or participants already exist and develop in their environment.
• The researcher does not intervene directly in the environment of the sample.
• This method studies the phenomena exactly as they occurred.

Types of non-experimental research

Non-experimental research can take the following forms:

Cross-sectional research : Cross-sectional research is used to observe and analyze the exact time of the research to cover various study groups or samples. This type of research is divided into:

• Descriptive: When values are observed where one or more variables are presented.
• Causal: It is responsible for explaining the reasons and relationship that exists between variables in a given time.

Longitudinal research: In a longitudinal study , researchers aim to analyze the changes and development of the relationships between variables over time. Longitudinal research can be divided into:

• Trend: When they study the changes faced by the study group in general.
• Group evolution: When the study group is a smaller sample.
• Panel: It is in charge of analyzing individual and group changes to discover the factor that produces them.

LEARN ABOUT: Quasi-experimental Research

When to use non-experimental research

Non-experimental research can be applied in the following ways:

• When the research question may be about one variable rather than a statistical relationship about two variables.
• There is a non-causal statistical relationship between variables in the research question.
• The research question has a causal research relationship, but the independent variable cannot be manipulated.
• In exploratory or broad research where a particular experience is confronted.

Advantages and disadvantages

Some advantages of non-experimental research are:

• It is very flexible during the research process
• The cause of the phenomenon is known, and the effect it has is investigated.
• The researcher can define the characteristics of the study group.

Among the disadvantages of non-experimental research are:

• The groups are not representative of the entire population.
• Errors in the methodology may occur, leading to research biases .

Non-experimental research is based on the observation of phenomena in their natural environment. In this way, they can be studied later to reach a conclusion.

Difference between experimental and non-experimental research

Experimental research involves changing variables and randomly assigning conditions to participants. As it can determine the cause, experimental research designs are used for research in medicine, biology, and social science. 

Experimental research designs have strict standards for control and establishing validity. Although they may need many resources, they can lead to very interesting results.

Non-experimental research, on the other hand, is usually descriptive or correlational without any explicit changes done by the researcher. You simply describe the situation as it is, or describe a relationship between variables. Without any control, it is difficult to determine causal effects. The validity remains a concern in this type of research. However, it’s’ more regarding the measurements instead of the effects.

LEARN MORE: Descriptive Research vs Correlational Research

Whether you should choose experimental research or non-experimental research design depends on your goals and resources. If you need any help with how to conduct research and collect relevant data, or have queries regarding the best approach for your research goals, contact us today! You can create an account with our survey software and avail of 88+ features including dashboard and reporting for free.

Create a free account

MORE LIKE THIS

2024 Presidential Election Polls: Harris vs. Trump

Nov 5, 2024

Your First Question Should Be Anything But, “Is The Car Okay?” — Tuesday CX Thoughts

QuestionPro or Qualtrics? Which Offers the Best Tool for Conducting 360-Degree Assessments?

Nov 4, 2024

Total Experience in Trinidad & Tobago — Tuesday CX Thoughts

Oct 29, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Non-Experimental Research

28 Overview of Non-Experimental Research

Learning objectives.

• Define non-experimental research, distinguish it clearly from experimental research, and give several examples.
• Explain when a researcher might choose to conduct non-experimental research as opposed to experimental research.

What Is Non-Experimental Research?

Non-experimental research  is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world).

Most researchers in psychology consider the distinction between experimental and non-experimental research to be an extremely important one. This is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, non-experimental research generally cannot. As we will see, however, this inability to make causal conclusions does not mean that non-experimental research is less important than experimental research. It is simply used in cases where experimental research is not able to be carried out.

When to Use Non-Experimental Research

As we saw in the last chapter , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable. It stands to reason, therefore, that non-experimental research is appropriate—even necessary—when these conditions are not met. There are many times in which non-experimental research is preferred, including when:

• the research question or hypothesis relates to a single variable rather than a statistical relationship between two variables (e.g., how accurate are people’s first impressions?).
• the research question pertains to a non-causal statistical relationship between variables (e.g., is there a correlation between verbal intelligence and mathematical intelligence?).
• the research question is about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions for practical or ethical reasons (e.g., does damage to a person’s hippocampus impair the formation of long-term memory traces?).
• the research question is broad and exploratory, or is about what it is like to have a particular experience (e.g., what is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and non-experimental approaches is generally dictated by the nature of the research question. Recall the three goals of science are to describe, to predict, and to explain. If the goal is to explain and the research question pertains to causal relationships, then the experimental approach is typically preferred. If the goal is to describe or to predict, a non-experimental approach is appropriate. But the two approaches can also be used to address the same research question in complementary ways. For example, in Milgram’s original (non-experimental) obedience study, he was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. However,  Milgram subsequently conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [1] .

Types of Non-Experimental Research

Non-experimental research falls into two broad categories: correlational research and observational research. 

The most common type of non-experimental research conducted in psychology is correlational research. Correlational research is considered non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable. More specifically, in correlational research , the researcher measures two variables with little or no attempt to control extraneous variables and then assesses the relationship between them. As an example, a researcher interested in the relationship between self-esteem and school achievement could collect data on students’ self-esteem and their GPAs to see if the two variables are statistically related.

Observational research  is non-experimental because it focuses on making observations of behavior in a natural or laboratory setting without manipulating anything. Milgram’s original obedience study was non-experimental in this way. He was primarily interested in the extent to which participants obeyed the researcher when he told them to shock the confederate and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of observational research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the researchers asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories).

Cross-Sectional, Longitudinal, and Cross-Sequential Studies

When psychologists wish to study change over time (for example, when developmental psychologists wish to study aging) they usually take one of three non-experimental approaches: cross-sectional, longitudinal, or cross-sequential. Cross-sectional studies involve comparing two or more pre-existing groups of people (e.g., children at different stages of development). What makes this approach non-experimental is that there is no manipulation of an independent variable and no random assignment of participants to groups. Using this design, developmental psychologists compare groups of people of different ages (e.g., young adults spanning from 18-25 years of age versus older adults spanning 60-75 years of age) on various dependent variables (e.g., memory, depression, life satisfaction). Of course, the primary limitation of using this design to study the effects of aging is that differences between the groups other than age may account for differences in the dependent variable. For instance, differences between the groups may reflect the generation that people come from (a cohort effect ) rather than a direct effect of age. For this reason, longitudinal studies , in which one group of people is followed over time as they age, offer a superior means of studying the effects of aging. However, longitudinal studies are by definition more time consuming and so require a much greater investment on the part of the researcher and the participants. A third approach, known as cross-sequential studies , combines elements of both cross-sectional and longitudinal studies. Rather than measuring differences between people in different age groups or following the same people over a long period of time, researchers adopting this approach choose a smaller period of time during which they follow people in different age groups. For example, they might measure changes over a ten year period among participants who at the start of the study fall into the following age groups: 20 years old, 30 years old, 40 years old, 50 years old, and 60 years old. This design is advantageous because the researcher reaps the immediate benefits of being able to compare the age groups after the first assessment. Further, by following the different age groups over time they can subsequently determine whether the original differences they found across the age groups are due to true age effects or cohort effects.

The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. But as you will learn in this chapter, many observational research studies are more qualitative in nature. In  qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s observational study of the experience of people in psychiatric wards was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semi-public room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256) [2] . Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable.  Figure 6.1 shows how experimental, quasi-experimental, and non-experimental (correlational) research vary in terms of internal validity. Experimental research tends to be highest in internal validity because the use of manipulation (of the independent variable) and control (of extraneous variables) help to rule out alternative explanations for the observed relationships. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Non-experimental (correlational) research is lowest in internal validity because these designs fail to use manipulation or control. Quasi-experimental research (which will be described in more detail in a subsequent chapter) falls in the middle because it contains some, but not all, of the features of a true experiment. For instance, it may fail to use random assignment to assign participants to groups or fail to use counterbalancing to control for potential order effects. Imagine, for example, that a researcher finds two similar schools, starts an anti-bullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” While a comparison is being made with a control condition, the inability to randomly assign children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying (e.g., there may be a selection effect).

Notice also in  Figure 6.1 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational (non-experimental) studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well-designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

• Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

A research that lacks the manipulation of an independent variable.

Research that is non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable.

Research that is non-experimental because it focuses on recording systemic observations of behavior in a natural or laboratory setting without manipulating anything.

Studies that involve comparing two or more pre-existing groups of people (e.g., children at different stages of development).

Differences between the groups may reflect the generation that people come from rather than a direct effect of age.

Studies in which one group of people are followed over time as they age.

Studies in which researchers follow people in different age groups in a smaller period of time.

Research Methods in Psychology Copyright © 2019 by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Non-Experimental Research: Designs, Characteristics, Types and Examples

The non-experimental research is one in which the variables of the study are not controlled or manipulated. To develop the research, the authors observe the phenomena to be studied in their natural environment, obtaining the data directly to analyze them later.

The difference between non-experimental and experimental research is that variables are manipulated in the latter and the study is carried out in controlled environments. So, for example, you experience gravity by intentionally dropping a stone from several heights.

On the other hand, in non-experimental research, researchers go, if necessary, to the place where the phenomenon to be studied happens. For example, to know the drinking habits of young people, surveys are conducted or observed directly as they do, but no drink is offered.

This type of research is very common in fields such as psychology, the measurement of unemployment rates, consumption studies or opinion polls. In general, these are pre-existing facts, developed under their own laws or internal rules

• 1.1 Differences with experimental designs
• 2 characteristics
• 3.1 Transverse or transectional design
• 3.2 Longitudinal design
• 4.1 Effects of alcohol
• 4.2 Opinion polls
• 4.3 School performance
• 5 References

Non-experimental research designs

As opposed to what happens with experimental research, in the non-experimental the variables studied are not deliberately manipulated. The way to proceed is to observe the phenomena to be analyzed as they are presented in their natural context.

In this way, there are no stimuli or conditions for the subjects that are being studied. These are found in their natural environment, without being transferred to any laboratory or controlled environment.

The existing variables are of two different types. The first are the independent calls, while the so-called dependents are a direct consequence of the previous ones. In this type of research, the relationships between causes and effects are investigated in order to draw valid conclusions.

Given that no exprofeso situations are created to investigate them, it can be affirmed that the non-experimental designs study the already existing situations developed under their own internal rules. In fact, another denomination that is given is that of investigations ex post facto ; that is, about facts fulfilled.

Differences with experimental designs

The main difference between both types of research is that in the experimental designs there is a manipulation of the variables by the researcher. Once the desired conditions have been created, the studies measure the effects of them.

For its part, in non-experimental investigations this manipulation does not exist, but rather the data is collected directly in the environment in which the events take place.

It can not be said that one method is better than the other. Each one is equally valid depending on what is going to be studied and / or on the perspective that the researcher wants to give to his work.

By its own characteristics, if the research is experimental it will be much easier to repeat it to ensure the results. However, the control of the environment makes some variables that may appear spontaneously more difficult to measure. It is just the opposite of what happens with non-experimental designs.

characteristics

As previously mentioned, the first characteristic of this type of research is that there is no manipulation of the variables studied.

Normally, these are phenomena that have already occurred and are analyzed a posteriori. Apart from this characteristic, other peculiarities present in these designs can be pointed out:

- Non-experimental research is widely used when, for ethical reasons (such as giving drink to young people), there is no option to conduct controlled experiments.

- No groups are formed to study them, but these are already pre-existing in their natural environments.

-The data is collected directly, and then analyzed and interpreted. There is no direct intervention on the phenomenon.

- It is very common that non-experimental designs are used in applied research, since they study the facts as they occur naturally.

- Given the characteristics presented, this type of research is not valid to establish unequivocal causal relationships.

Transverse or transectional design

This type of non-experimental research design is used to observe and record the data at a specific time and, by its very nature, unique. In this way, the analysis is focused on the effects of a phenomenon that occurs at a particular time.

As an example, we can mention the study of the consequences of an earthquake on the housing in a city or the school failure rates in a given year. You can also take more than one variable, turning the study into a more complex one.

The transversal design allows to cover diverse groups of individuals, objects or phenomena. At the time of developing them, they can be divided into two different groups:

Descriptive

The objective is to investigate those incidents and their values, in which one or more variables appear. Once the data is obtained, a description of them is simply made.

In these designs, we try to establish the relationships between several variables that have occurred at a given moment. These variables are not described one by one, but rather they try to explain how they are related.

Longitudinal design

Contrary to what happens with the previous design, in the longitudinal the researchers intend to analyze the changes suffered by certain variables over time. You can also investigate how the relationships between these variables evolve during this period.

To achieve this goal it is necessary to collect data at different time points. There are three types within this design:

They study the changes that happen in some population in general.

Of group evolution

The subjects studied are smaller groups or subgroups.

Similar to the previous ones but with specific groups that are measured at all times. These investigations are useful to analyze the individual changes together with the group, allowing to know what element has produced the changes in question.

In general terms, these designs are prepared for the study of events that have already happened and, therefore, it is impossible to control the variables. They are very frequent in statistical fields of all kinds, both to measure the incidence of some factors and for opinion studies.

Effects of alcohol

A classic example of non-experimental research is studies on the effects of alcohol on the human body. Since it is not ethical to give drink to the subjects studied, these designs are used to obtain results.

The way to achieve this would be to go to the places where alcohol is habitually consumed. There is measured the degree that this substance reaches in blood (or you can take data from the police or a hospital). With this information, we will proceed to compare the different individual reactions, extracting the conclusions about it.

Opinion polls

Any survey that tries to measure the opinion of a certain group on a topic is done through non-experimental designs. For example, electoral polls are very common in most countries.

School performance

It would only be necessary to collect the statistics of the results of the school children offered by the schools themselves. If, in addition, you want to complete the study, you can search for information on the socioeconomic status of the students.

Analyzing each data and relating them to each other, a study is obtained about how the socioeconomic level of families affects the performance of school children.

• APA rules. Non-experimental research - What they are and how to elaborate them. Retrieved from normasapa.net
• EcuREd. Non-experimental research. Retrieved from ecured.cu
• Methodology2020. Experimental and non-experimental research. Retrieved from metodologia2020.wikispaces.com
• Rajeev H. Dehejia, Sadek Wahba. Propensity Score-Matching Methods for Nonexperimental Causal Studies. Retrieved from business.baylor.edu
• ReadingCraze.com. Research Design: Experimental and Nonexperimental Research. Retrieved from readingcraze.com
• Reio, Thomas G. Nonexperimental research: strengths, weaknesses and issues of precision. Retrieved from emeraldinsight.com
• Wikipedia. Research design. Retrieved from en.wikipedia.org

Recent Posts

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Overview of Non-Experimental Research

Rajiv S. Jhangiani; I-Chant A. Chiang; Carrie Cuttler; and Dana C. Leighton

Learning Objectives

• Define non-experimental research, distinguish it clearly from experimental research, and give several examples.
• Explain when a researcher might choose to conduct non-experimental research as opposed to experimental research.

What Is Non-Experimental Research?

Non-experimental research  is research that lacks the manipulation of an independent variable. Rather than manipulating an independent variable, researchers conducting non-experimental research simply measure variables as they naturally occur (in the lab or real world).

Most researchers in psychology consider the distinction between experimental and non-experimental research to be an extremely important one. This is because although experimental research can provide strong evidence that changes in an independent variable cause differences in a dependent variable, non-experimental research generally cannot. As we will see, however, this inability to make causal conclusions does not mean that non-experimental research is less important than experimental research. It is simply used in cases where experimental research is not able to be carried out.

When to Use Non-Experimental Research

As we saw in the last chapter , experimental research is appropriate when the researcher has a specific research question or hypothesis about a causal relationship between two variables—and it is possible, feasible, and ethical to manipulate the independent variable. It stands to reason, therefore, that non-experimental research is appropriate—even necessary—when these conditions are not met. There are many times in which non-experimental research is preferred, including when:

• the research question or hypothesis relates to a single variable rather than a statistical relationship between two variables (e.g., how accurate are people’s first impressions?).
• the research question pertains to a non-causal statistical relationship between variables (e.g., is there a correlation between verbal intelligence and mathematical intelligence?).
• the research question is about a causal relationship, but the independent variable cannot be manipulated or participants cannot be randomly assigned to conditions or orders of conditions for practical or ethical reasons (e.g., does damage to a person’s hippocampus impair the formation of long-term memory traces?).
• the research question is broad and exploratory, or is about what it is like to have a particular experience (e.g., what is it like to be a working mother diagnosed with depression?).

Again, the choice between the experimental and non-experimental approaches is generally dictated by the nature of the research question. Recall the three goals of science are to describe, to predict, and to explain. If the goal is to explain and the research question pertains to causal relationships, then the experimental approach is typically preferred. If the goal is to describe or to predict, a non-experimental approach is appropriate. But the two approaches can also be used to address the same research question in complementary ways. For example, in Milgram’s original (non-experimental) obedience study, he was primarily interested in one variable—the extent to which participants obeyed the researcher when he told them to shock the confederate—and he observed all participants performing the same task under the same conditions. However,  Milgram subsequently conducted experiments to explore the factors that affect obedience. He manipulated several independent variables, such as the distance between the experimenter and the participant, the participant and the confederate, and the location of the study (Milgram, 1974) [1] .

Types of Non-Experimental Research

Non-experimental research falls into two broad categories: correlational research and observational research. 

The most common type of non-experimental research conducted in psychology is correlational research. Correlational research is considered non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable. More specifically, in correlational research , the researcher measures two variables with little or no attempt to control extraneous variables and then assesses the relationship between them. As an example, a researcher interested in the relationship between self-esteem and school achievement could collect data on students’ self-esteem and their GPAs to see if the two variables are statistically related.

Observational research  is non-experimental because it focuses on making observations of behavior in a natural or laboratory setting without manipulating anything. Milgram’s original obedience study was non-experimental in this way. He was primarily interested in the extent to which participants obeyed the researcher when he told them to shock the confederate and he observed all participants performing the same task under the same conditions. The study by Loftus and Pickrell described at the beginning of this chapter is also a good example of observational research. The variable was whether participants “remembered” having experienced mildly traumatic childhood events (e.g., getting lost in a shopping mall) that they had not actually experienced but that the researchers asked them about repeatedly. In this particular study, nearly a third of the participants “remembered” at least one event. (As with Milgram’s original study, this study inspired several later experiments on the factors that affect false memories).

Cross-Sectional, Longitudinal, and Cross-Sequential Studies

When psychologists wish to study change over time (for example, when developmental psychologists wish to study aging) they usually take one of three non-experimental approaches: cross-sectional, longitudinal, or cross-sequential. Cross-sectional studies involve comparing two or more pre-existing groups of people (e.g., children at different stages of development). What makes this approach non-experimental is that there is no manipulation of an independent variable and no random assignment of participants to groups. Using this design, developmental psychologists compare groups of people of different ages (e.g., young adults spanning from 18-25 years of age versus older adults spanning 60-75 years of age) on various dependent variables (e.g., memory, depression, life satisfaction). Of course, the primary limitation of using this design to study the effects of aging is that differences between the groups other than age may account for differences in the dependent variable. For instance, differences between the groups may reflect the generation that people come from (a cohort effect ) rather than a direct effect of age. For this reason, longitudinal studies , in which one group of people is followed over time as they age, offer a superior means of studying the effects of aging. However, longitudinal studies are by definition more time consuming and so require a much greater investment on the part of the researcher and the participants. A third approach, known as cross-sequential studies , combines elements of both cross-sectional and longitudinal studies. Rather than measuring differences between people in different age groups or following the same people over a long period of time, researchers adopting this approach choose a smaller period of time during which they follow people in different age groups. For example, they might measure changes over a ten year period among participants who at the start of the study fall into the following age groups: 20 years old, 30 years old, 40 years old, 50 years old, and 60 years old. This design is advantageous because the researcher reaps the immediate benefits of being able to compare the age groups after the first assessment. Further, by following the different age groups over time they can subsequently determine whether the original differences they found across the age groups are due to true age effects or cohort effects.

The types of research we have discussed so far are all quantitative, referring to the fact that the data consist of numbers that are analyzed using statistical techniques. But as you will learn in this chapter, many observational research studies are more qualitative in nature. In  qualitative research , the data are usually nonnumerical and therefore cannot be analyzed using statistical techniques. Rosenhan’s observational study of the experience of people in psychiatric wards was primarily qualitative. The data were the notes taken by the “pseudopatients”—the people pretending to have heard voices—along with their hospital records. Rosenhan’s analysis consists mainly of a written description of the experiences of the pseudopatients, supported by several concrete examples. To illustrate the hospital staff’s tendency to “depersonalize” their patients, he noted, “Upon being admitted, I and other pseudopatients took the initial physical examinations in a semi-public room, where staff members went about their own business as if we were not there” (Rosenhan, 1973, p. 256) [2] . Qualitative data has a separate set of analysis tools depending on the research question. For example, thematic analysis would focus on themes that emerge in the data or conversation analysis would focus on the way the words were said in an interview or focus group.

Internal Validity Revisited

Recall that internal validity is the extent to which the design of a study supports the conclusion that changes in the independent variable caused any observed differences in the dependent variable.  Figure 6.1 shows how experimental, quasi-experimental, and non-experimental (correlational) research vary in terms of internal validity. Experimental research tends to be highest in internal validity because the use of manipulation (of the independent variable) and control (of extraneous variables) help to rule out alternative explanations for the observed relationships. If the average score on the dependent variable in an experiment differs across conditions, it is quite likely that the independent variable is responsible for that difference. Non-experimental (correlational) research is lowest in internal validity because these designs fail to use manipulation or control. Quasi-experimental research (which will be described in more detail in a subsequent chapter) falls in the middle because it contains some, but not all, of the features of a true experiment. For instance, it may fail to use random assignment to assign participants to groups or fail to use counterbalancing to control for potential order effects. Imagine, for example, that a researcher finds two similar schools, starts an anti-bullying program in one, and then finds fewer bullying incidents in that “treatment school” than in the “control school.” While a comparison is being made with a control condition, the inability to randomly assign children to schools could still mean that students in the treatment school differed from students in the control school in some other way that could explain the difference in bullying (e.g., there may be a selection effect).

Notice also in  Figure 6.1 that there is some overlap in the internal validity of experiments, quasi-experiments, and correlational (non-experimental) studies. For example, a poorly designed experiment that includes many confounding variables can be lower in internal validity than a well-designed quasi-experiment with no obvious confounding variables. Internal validity is also only one of several validities that one might consider, as noted in Chapter 5.

• Milgram, S. (1974). Obedience to authority: An experimental view . New York, NY: Harper & Row. ↵
• Rosenhan, D. L. (1973). On being sane in insane places. Science, 179 , 250–258. ↵

A research that lacks the manipulation of an independent variable.

Research that is non-experimental because it focuses on the statistical relationship between two variables but does not include the manipulation of an independent variable.

Research that is non-experimental because it focuses on recording systemic observations of behavior in a natural or laboratory setting without manipulating anything.

Studies that involve comparing two or more pre-existing groups of people (e.g., children at different stages of development).

Differences between the groups may reflect the generation that people come from rather than a direct effect of age.

Studies in which one group of people are followed over time as they age.

Studies in which researchers follow people in different age groups in a smaller period of time.

Overview of Non-Experimental Research Copyright © by Rajiv S. Jhangiani; I-Chant A. Chiang; Carrie Cuttler; and Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Student Portal: UNE Online Research Toolkit -

Brightspace U-Online UNE Email

Research Toolkit

• Action Research
• Ethnography
• Narrative Study
• Interpretive Phenomenological Analysis
• Transcendental Phenomenology
• Experimental Design

Non-Experimental Design

• Quasi-Experimental Design
• Mixed Methods Design
• Citation Tools
• Presentation Tips
• Video Tutorials (MS Office Beginners Guides)
• Video Tutorials (MS Office Intermediate & Advanced)
• Visual Organizers
• Writing, Editing, and Transcription Tips
• UNE Approved: Descriptive and Survey Research Tools
• CITI Training
• UNE IRB Website

Studies using descriptive design do not include a control group for comparison.

Not all research is about measuring the effects of an intervention on one group compared to a group that did not receive the intervention. There is another class of quantitative research design called non-experimental research.  These research designs can be used to show relationships between variables (correlational design). Non-experimental research can also be used to study he existence or incidence of a phenomenon (descriptive design). Note that in a non-experimental design, the independent variable is not controlled.

Correlational retrospective design. Research studies in this category examine how an event in the past may have an effect in the present, for example, presence of cancer and pulmonary disease rates among residents of New York City after the 9/11 bombings.

Correlational prospective design. In a prospective design, researchers believe that a phenomenon may have a future effect on the population of interest. For example, is the existence of lead-based paint in the home correlated to lower education scores? Newborns living in homes with lead-based paint would be followed by the researches and tested on some regular basis to compare test scores.

Descriptive correlational studies demonstrate the relationship among variables without going as far as showing cause and effect. Prevalence and incidence studies are examples of this type of research.  In epidemiology, a prevalence study is used to study the saturation of a condition whereas an incidence study is used to study the rate of new cases in the population.

Statistical analysis

Statistical measures of correlation will depend on the data type.  Incidence and prevalence are measured by a defined formula for rate.

Strengths and limitations

Non-experimental research lacks the reliability and validity of quasi-experimental and experimental research designs. However, findings from non-experimental research is the first step in determining whether an experimental design is called for.

RUM leads to noise: the significance of finding the sources of variability between experimental runs

• Original Research
• Open access
• Published: 04 November 2024
• Volume 204 , article number  147 , ( 2024 )

Cite this article

You have full access to this open access article

• Michael Vogrin   ORCID: orcid.org/0000-0003-2750-8595 1 &
• Jan Willem Koten   ORCID: orcid.org/0000-0003-3661-9238 1  

25 Accesses

Explore all metrics

Scientists typically run experiments many times to find general patterns over multiple specific runs. The results of those runs vary, and the variance is often simply referred to as “noise”. We claim that it is highly important to separate the components that contribute to noise and to recognize to which degree they contribute to it. Consideration of the relative contributions of R (randomness), U (uncontrolled variables), and M (measurement error) helps to interpret data and can help to improve experimental designs. We explain this using a hypothetical example and point out that assumptions of what causes variability in the results of experiments are often made implicitly. Further, we demonstrate our point by showing how it can change the interpretation of real data. Because of a lack of explicit discussion of underlying assumptions, it is possible that sources of noise are misidentified to be either existent or non-existent. This can happen if, for example, measurement error is assumed when there is none, an assumption that would mask a real effect that could deserve further study. Despite these factors, the contribution of different factors to the overall noise is rarely considered, hampering scientific progress.

Similar content being viewed by others

On the correct interpretation of p values and the importance of random variables.

Randomness and the Games of Science

Heterogeneity and Causality

Avoid common mistakes on your manuscript.

1 Introduction

Experimental science is dependent on reproducibility (McNutt, 2014 , Gundersen 2021 ), for if outcomes vary drastically, we do not learn anything meaningful from them. But why would results not be reproducible in a world with unchanging natural laws governed by cause and effect? Three reasons come to mind: randomness (R), uncontrolled variables (U) and measurement error (M). In the following, three hypothetical experiments are presented to illustrate these factors.

Whenever sampling is involved, part of the variety of some target variable between different groups or points of time can be explained by the simple fact that there is variation within the population from which the sample is drawn. To illustrate how randomness might reduce reproducibility, think of a deck of cards. Drawing a hand of seven and adding up the numbers will likely have a similar, but not the same result as when doing it again after shuffling. This is because there is variation within the deck of cards (the different card values), and the sampling (drawing a hand of seven cards) is random (ensured through the shuffling). In this example, we assume that there is no measuring error, i.e., the determination and summation of the card values are without mistake. Also, there are no other uncontrolled variables, meaning that factors such as the weather and others do not matter. Here, all the variation results from randomness.

Whenever variables that are not measured influence the outcome, there will be significant impact of unconsidered variables. For example, the amount of money spent on ice cream by John Doe in May can be recorded each year. However, this amount will not always be the same, reducing the reproducibility. This reduction is not explained by randomness or measuring error (as the amount of money can be tracked very accurately using bank statements), but instead by unconsidered factors. For example, if it is significantly hotter in May 2024 than it is in May 2014, John Doe might buy more ice cream. Therefore, temperature is a moderating factor that is not controlled for in this setup and will influence the variance between observations.

Whenever the target variables are not trivial to access, there will be some degree of measuring error. Sometimes this is the only source of error. For example, a bowl of rice will not change in weight significantly over the course of one hour. However, the weight in grams may not be perfectly reproducible when measuring once at 10:00 and once at 11:00 with a cheap kitchen scale. The variation cannot be attributed to randomness or uncontrolled variables.

In practice, scientific experiments are subject to all three factors, randomness, uncontrolled variables, and measurement error. Scientists are so used to this that they subsume the effects of all three factors as “ noise ” in the data, which is rarely discussed further (e.g. Sripada et al., 2016 , for exceptions see e.g. Andrade, 2023 , Scales & Snieder, 1998 ). What is more, the three factors seem to blur together occasionally. Consider a drug study in which an experimental group receives a drug to increase longevity is compared to a control group who receives a placebo. It could be the case that the control group lives longer on average, despite the drug working perfectly fine. This can happen if the people in the control group just happen to exercise more, eat better, have genes related to longevity, simply get into fewer accidents, and so on. These would all be uncontrolled variables. However, the fact that one group– in this scenario the control group– disproportionally has those participants that show these characteristics is due to randomness itself. Therefore, despite possible inclusion criteria applied in studies that change who participates and try to eliminate the influence of chance, it cannot be reduced to zero. In this context it is also important to consider what “randomness” means. For some authors, it means that every outcome is equiprobable (equally likely). Other authors make a point of explicitly stating that this is not a necessary condition for randomness:

“[…] the word random should not be mistaken for equiprobable (i.e. having the same probability” (Taleb:42).

Consider rolling a fair die. By definition, the outcome of the roll is both random and equiprobable, and both kinds of authors agree. However, if the event of interest is rolling exactly a six, then this is still a random event for most people, even though the two different outcomes (not a six or six) are not equiprobable. So, to align with everyday intuitions and because this conception of randomness maps better onto our context, we view events as random also if they are not equiprobable.

To return to the broader picture, it is tremendously helpful to split up this ominous noise into its constituents, and think about the contributions of those constituents towards noise. This is because once noise is split up into its constituents, they can be understood and - at least in principle - be reduced. Figure  1 illustrates this for the hypothetical experiments.

The RUM-triangle shows that what we call “noise” is really three mostly independent phenomena: measuring error (M), uncontrolled variables (U), and randomness (R). The three experiments mentioned have one clear source of noise, while real science (represented by the microscope) usually has a not clearly defined mix out of all three factors

However, the crucial aspect of this analysis is not that the contribution of the three factors should be better understood, so that experiments can be designed a little better. Although this can be a useful ambition sparked by our analysis, the deeper philosophical and methodological implications are what we want to focus on here. We suggest that there is a rift between data and interpretation and that the seldomly addressed assumptions about where noise comes from are used as a flimsy bridge to cross this rift.

2 Background assumptions determine methods

Scientists define concepts, measure variables, record and interpret this data. The interpretation of data is done relative to the rest of the literature, has an aspect of subjectivity to it, and can be influenced by different biases (Järvinen et al., 2014 ) or the perceived level of noise (Harvey et al. 1997 ). The interpretation of one scientist may differ from the one of another. Nevertheless, this does not seem to be a problem, because there is a clear distinction between “the data” and “the interpretation”. However, rarely raw data is interpreted. Instead, data is processed: means, correlations, or “statistically significant differences” are calculated. And this is where it becomes difficult to say what is right and what is wrong, since what can be reasonably done to data, is not within the data. In the following, we demonstrate this on a hypothetical example comparing two measurements that could produce the same data, but lead to different interpretations because of different positions of the experiments in the RUM-triangle.

The human genome of a person is, ignoring epigenetic modifications, unchanging over the human’s lifespan (McKinnon, 2013 ). This means that the measurement of the genetic code of a person in January 2024 should be exactly the same as another measurement in April 2024. However, this is not the case. Because of the way it is measured, small errors can occur. This means that in a sample of genetic twins, the test-retest reliability might be 95% instead of 100%, with the remaining 5% being attributed to measurement error (Beck et al., 2019 ). Then, in the same sample, and with the same machines, one could find that the genomes of the twins correlate to 97%. This is strange, since the data would then imply that twin A is more similar to his twin B, than twin A is similar to himself at another point of time. Therefore, in the same way as the missing 5% in reliability for the perfect 100% test-retest reliability were attributed to measuring error, these 2% of “overperformance” found in the correlations of the twins need to be explained. Here, a decision is made. Based on what we know (or think to know) about genomes, we decide that the correlation between twins cannot possibly be higher than the correlation between one person with themselves at two different points of time. This correction makes sense if the genome indeed does not change significantly. Nevertheless, in other cases such corrections are not meaningful. If one compares results of a blood test, then it is very easily possible that person A does not correlate perfectly with themself at a later point of time, and at the same time correlates almost perfectly with someone else. Figure  2 illustrates this situation.

An example of how background assumptions inform analyses. In the case on the left, the test-retest correlation for A is at a high 95%, but measuring A and comparing him with his genetic twin B results in an even higher 97% correlation. The 2% “overperformance” is attributed to measurement error, because one cannot have a higher match with someone else than with oneself. However, when measuring metabolites in the blood that vary over time, no such correction is needed

However, if and when such a correction should be done, is not within the data. It depends on implicit assumptions, which we call “background assumptions” or “background knowledge” (Thagard & Nisbett, 1982 ). In this case those assumptions are that the genome does not fluctuate and perfect measurements over time would always give the same results, while some target parameters assessed while sampling blood indeed fluctuate. Therefore, the position in the RUM-triangle determines, at least in part, which analyses make sense. In practice, measurement error and chance variance can occur simultaneously, making it difficult to estimate the measurement error (Bland & Altman, 1996 , Beckerman et al. 2001 ).

2.1 The problem: being lost in the RUM-triangle

A cynical reader might think that we are not on the way towards the introduction of any sort of problem whatsoever. After all, it makes total sense to attribute findings that are not consistent with preexisting knowledge to measuring error, especially if it is known that measuring error exists, and other sources of error can be excluded. But what if they cannot be excluded?

The correction done in the analysis of the genome presupposes that there is no significant randomness and that there are no uncontrolled variables contributing to the variance observed. Proponents of the correction would place the experiment into the M corner of the triangle, postulating that the biggest, perhaps the only, source of error is measurement error. This is the perceived position in the triangle. However, it is possible that one is wrong with this perception and that the actual position in the triangle is somewhere else, see Fig.  3 . This is where the cynic also has to admit that things become problematic.

If the real position and the assumed position of an experimental design are significantly different, there will be a large degree of attribution error

After all, it could be the case that the assumed measuring error does not exist, and that what is assumed to be an error is an actual phenomenon. Such misattribution can have considerable consequences that go beyond having “noisy data”. If measuring error is wrongly attributed to be the (only) source of error, experiments will be wrongly interpreted! Just for the sake of argument, let us assume that there is absolutely no measurement error in any of the procedures used to do the genome sequencing. Then, the correction made as shown in Fig.  2 is done wrongfully. It may mask an interesting effect. For example, it could be that what is measured fluctuates during the day, similar to metabolites in the blood, and that higher correlations with a twin than with oneself at a later point of time are not a mystery to be “explained away” by measuring error, but an interesting fact to be researched in further studies. This is exemplified in the field of psychology: people when asked about sensitive topics often do not report the truth, even if surveys are anonymized. One might call the difference between their actual opinions and the reported opinions simply “measurement error” (e.g. Blattman et al., 2016 ) or study this phenomenon as a result of an uncontrolled variable and name it “social desirability” (Nederhof, 1985 ; Chung & Monroe, 2003 ; Grimm, 2010 ).

2.2 A practical example: setup

We present a practical example of how one could determine the position in the RUMtriangle, and how this position influences how data is interpreted. In an experiment, reaction time was measured in two ways: on the one hand, brain activity was measured using an fMRI-scanner, and on the other hand, the same participant pushed a button and the reaction time was measured. The experiment was repeated so that there are two measurements for both methods. As expected, there is some variability in what is measured, but overall, the experiment is reproducible. In practice, most scientist would call the experiment a success and spend little time contemplating why there is residual variability in the results. We want to show why it is worthwhile to think about the different sources of variability.

As mentioned, variability (noise) can come from randomness, uncontrolled variables, and measurement error. Because the participant is the same for both ways of measurement and all experimental runs, we want to ignore randomness for now. The possible sources for error are therefore uncontrolled variables and measurement error. Because the FRMI-scanner is a complicated machine producing slightly inhomogeneous magnetic fields that interact with the physiology of the individual under study, we assume some measurement error. We can tentatively place the experimental design of measuring reaction times for the specific task using the scanner in the RUM-triangle, close to the M-corner. In addition, we can apply denoising to the fRMI data, which reduces the measuring error, moving it closer to the U-corner, see Fig.  4 . Now, how can we verify if this tentative placement is more or less correct? Here, the second way of measuring the system is informative. Measuring the reaction time by letting the participant push the button has negligible measuring error, as the button reacts sensitive and reliable to the movement of the participant and the signal is conducted via cables to the computer that acts as a recording device. In this setup, the variability of the behavior shown by the same person is lower. This means that measurement error indeed contributes to the variability of the results when using the FRMI-scanner. However, the variability when measuring the button pushes is still not zero, meaning that there is a source of noise.

Because we assume the measurement error to be negligible, and exclude randomness since no sampling is done, uncontrolled variables must be at play, see Fig.  4 .

Three ways of measuring the same phenomenon have different sources of noise, and different levels of noise overall, influencing the reproducibility

In Fig.  5 , we present exemplary results of a real fMRI experiment in which participants were presented with visual stimuli that triggered working memory related neural responses in the brain. These responses lead to motor responses that reflected the time taken to complete the cognitive task. The cognitive task phases were interrupted by periods of cognitive rest in a regular fashion, causing the fMRI time courses extracted from the right mid frontal cortex to display cyclic behavior. Peaks in brain activity indicated cognitive action, while lows in brain activity indicated cognitive rest.

In a first experiment presented in Fig.  5 the test-retest reliability of the raw fMRI time courses was lower compared to the denoised fMRI time courses, indicating that fMRI experiments involve measurement errors that can be controlled for. As behavior is a result of neural processes, it is reasonable to assume that reliable neural responses result in reliable reaction time responses. In our experiment, the test-retest reliability of the fMRI time courses and response times was highly similar, suggesting a causal relationship between neural reliability and response time reliability. As response times contain little measurement error, one might speculate about the source of the leftover variability. Possibly it is due to uncontrolled variables affecting the reliability of neural processes in the right middle frontal cortex and the closely related behavior, which could be linked to internal neural processes within the brain or external stimuli from the individual’s environment that affect the brain.

Time courses extracted from the right middle frontal cortex for a test and retest run are shown in the top panel. The middle panel shows time courses that were detrended and denoised using white matter, ventricle, and head motion time courses. The bottom panel reports 48 response times for a working memory task involving a distracting number Stroop task, where participants had to memorize two target stimuli positions, identify the physical larger of two presented numbers, and determine if the location of a new stimulus matched the location of the memorized items. Note that as, from A over B to C, the contribution of measurement error decreases, and reproducibility increases

The overall findings are challenged when analyzing data from another individual presented in Fig.  6 . In this case, the reliability of the time courses again improves when measurement errors are removed (or at least reduced). However, the reliable brain activity observed does not necessarily translate into reliable response time behavior. As the very low response time reliability is most likely not due to measurement error, we have to conclude that the right middle frontal cortex indeed responds reliably to the visual stimuli presented but that theses time course fluctuations may not accurately reflect the neural processes involved in task-solving in this scenario, as time course behavior should ideally correlate with response time behavior, which appears unreliable. We have to conclude that some unknown variables within the brain most likely not located in the mid frontal cortex may be the actual cause of poor response time reliability. Further, this may also be the case for individual 1 (shown in Fig.  5 ), despite high correlations, high reproducibility, and background assumptions that make causality plausible. Taking some distance by comparing experiments one and two we have to conclude that the inconsistencies between the observations of the respective experiment are due to sampling error. In this case taking sampling error into account leads to a different interpretation of the nature of the uncontrolled variables as they occur in experiment one and two. However, this conclusion only makes sense under the assumption that measurement error is indeed lower in the paradigm that measured the response time by letting participants push a button (method C). This seems to be the case, as shown by the higher reproducibility of results coming from this paradigm (see Fig.  5 ), but what really is important here is this: one needs to think about where variability between subjects and/or different runs with the same subject comes from!

The same methods as described in Fig.  5 were used here. Despite good reproducibility using measurement methods A and B, the reproducibility is low in C. Assuming that brain states correlate strongly with behavior, this finding needs an explanation. However, the fitness of the explanation is dependent on the position of the experiment in the RUM-triangle

3 Repeating experiments solves the issue– or does it?

It seems that it is not all that important to recognize where noise comes from. After all, experiments can simply be repeated enough times, so that noise disappears, and the signal comes through. In practice, this can be tremendous effort. Instead of doing that, one could evaluate the position of the experiment in the RUM-triangle and eliminate the biggest contribution to noise. In an ideal scenario, one finds that measurement error is the biggest contributor, improves the measuring device, and in turn increases data quality without having to increase the number of experimental runs or the sample size. This is only possible if one thinks about the contributions to the noise in the data deeply, and therefore a practical advantage that our framework can lead to. More importantly, increasing the number of experimental runs to increase data quality only works given two conditions: first, the focus of the analysis must be on data that can be averaged. Often, we are interested in population means: to what degree does a certain medication decrease blood pressure on average ? What is the mean time it takes a customer to make the decision to buy in a specific situation? However, not all scientific enquiry is focussed on data that can be gathered by averaging many data points. Sometimes, it is important to record the extremes of the distribution. In such cases, more repetitions do not necessarily make the conclusions more reliable. The second condition is by far the more important one. Reducing the influence of noise on the observed signal by increasing the numbers of observations is only possible if the noise is not systematic.

3.1 Random noise vs. systematic noise

According to the RUM schema, there are three sources of noise. In addition, all noise can be categorized as either systematic noise or random noise– albeit the terms “systematic error” and “random error” are used more widely (e.g. Krippendorf 1970 , Hibbert, 2007 ). Here, random error refers to deviations that do not follow a pattern, while systematic error can be– at least in principle– traced back to one or more specific causes. Random error can be reduced by increasing the number of experimental runs. For example, if it is equally likely that a scientist or a measurement device underreports or overreports the measured variable, random error is in play. Increasing the number of experimental trials will increase the validity of the mean results, as the instances of underreporting and overreporting even out. This is easy to see: tossing a fair coin 10 times has a chance of about 17% to result in 7 or more heads. Meanwhile, tossing a fair coin 20 times only has a chance of about 6% to bear a similarly warped result (at least 14 heads).

In contrast, systematic error does not even out in the same way, even when increasing the number of experimental runs. For example, a biased measuring system might be too sensitive to what it is supposed to measure. Instead of the real value, it tends to overreport. Now, more repetitions do not make the result more reliable, because the errors do not average out. And this is exactly why reflecting on the sources of error is so important. Many researchers think that more repetitions make the reported means more reliable, and therefore have higher confidence in their results. But this is only true if random error contributes significantly more to the total error than systematic error.

4 Discussion

For scientists, it is so obvious that experiments need to be repeated, so that seldomly anyone thinks about the reasons why experiments need to be repeated. Upon reflection of those reasons, one realizes that “noise” comes in different forms: randomness, uncontrolled variables, and measuring error. Perhaps because none of those sources for error is a priori worse than the other, little effort is put into estimating - let alone measuring - which source contributes to which degree to the variability of outcomes for a given experiment. Interestingly, the entanglement of the different causes for noise is sometimes not a problem at all. In empirical studies (as well as in modelling studies) noise may be accepted or even introduced on purpose. The reasoning for this is that relevant effects should be stable enough and detectable despite such noise.

However, we showed that assumptions about where error comes from influence how data is interpreted. How data should be interpreted depends on factors outside the data. This is especially easy to see if one considers measuring error: if measuring error is the known - and only - cause of variations between subjects (or measurements of the same subject at different points of time), then these variations are of little scientific interest. Investigating them would tell us more about the measuring procedures or devices than about the rest of the universe. However, if measurement error can be excluded, then the variations between the measurements (could) become a subject for experimental investigation in themselves. However, to decide which is the case, one needs to know the position of the experimental design in the RUM-triangle. Rarely do we know this position with great accuracy, sometimes we can estimate it roughly, and in most cases this position is not considered explicitly at all. Knowing this position allows us to improve the experimental designs by actively trying to reduce the source of error that is the most influential. However, there is a danger of incorrectly diagnosing the position of the experimental design in the RUM-triangle: one might wrongfully exclude hypotheses if one is convinced that variation in measurements is necessarily due to measuring error. So, what can be done about this?

Clearly, none of the mentioned problems arise when the exact contributions of each of the factors are known. However, it is not obvious how these contributions for variation in a specific experiment could be accessed. Doing so reliably would necessitate either infallible theoretical background knowledge, or experiments about the experiment in question. The former can never be reasonably assumed as scientific theories need to be falsifiable (Popper, 2014 ). The latter can lead to an infinite regress, since the experiments to measure the sources of errors for the experiment will have variability in themselves, of which the contributions of randomness, uncontrolled variables, and measurement error are unclear. Regarding the issue of measurement error, this problem was recognized by Collins as the “experimenter’s regress” (Collins, 1992 ). More generally, Collins suggests that to know if an experiment has been well conducted, one needs to know whether it gives rise to the correct outcome. But to know this outcome in the first place, a well-conducted experiment is needed (Collins, 2016 ). However, given background knowledge, one can in fact use the same thing with different methods. Simpler and more direct methods, such as the pressing of a button in our example to measure the response time reliability, can illuminate the degree of measuring error of more indirect methods, such as measuring brain states while doing the same task. Further, in some scenarios it is unclear where to draw the line between, for example, uncontrolled variables and randomness: according to Rasch (as cited in Arocha, 2021 ) one could see variations in behavior not as a result of uncontrolled variables, but as a result of randomness, perhaps caused by noise in the nervous system (Faisal et al., 2008 ). Further, it is “perfectly possible that a properly applied random process might “by chance” produce a division between control and experimental groups that is significantly skewed with respect to some uncontrolled prognostic factor that in fact plays a role in therapeutic outcome” (Worrall, 2002 , p.322). Considering these challenges, seemingly the best we can do is to estimate the position of our experimental designs in the RUM-triangle as precisely as possible, and be open to the possibility that this tentative placement could be wrong. The estimation must neither be final nor infallible in order to provide value.

Andrade, C. (2023). Understanding statistical noise in research: 2. Noise in clinical trials and observational studies. Indian Journal of Psychological Medicine , 45 (2), 198–200.

Article   Google Scholar  

Arocha, J. F. (2021). Scientific realism and the issue of variability in behavior. Theory & Psychology , 31 (3), 375–398.

Beck, J. J., Hottenga, J. J., Mbarek, H., Finnicum, C. T., Ehli, E. A., Hur, Y. M., & Davies, G. E. (2019). Genetic similarity assessment of twin-family populations by custom-designed genotyping array. Twin Research and Human Genetics , 22 (4), 210–219.

Beckerman, H., Roebroeck, M. E., Lankhorst, G. J., Becher, J. G., Bezemer, P. D., & Verbeek, A. L. M. (2001). Smallest real difference, a link between reproducibility and responsiveness. Quality of Life Research , 10 , 571–578.

Bland, J. M., & Altman, D. G. (1996). Statistics notes: Measurement error. Bmj , 312 (7047), 1654.

Blattman, C., Jamison, J., Koroknay-Palicz, T., Rodrigues, K., & Sheridan, M. (2016). Measuring the measurement error: A method to qualitatively validate survey data. Journal of Development Economics , 120 , 99–112.

Chung, J., & Monroe, G. S. (2003). Exploring social desirability bias. Journal ofBusiness Ethics , 44 , 291–302.

Google Scholar  

Collins, H. (1992). Changing order: Replication and induction in scientific practice . University of Chicago Press.

Collins, H. (2016). Reproducibility of experiments: Experimenters’ regress, statistical uncertainty principle, and the replication imperative. Reproducibility: Principles, problems, practices, and prospects, 65–82.

Faisal, A. A., Selen, L. P., & Wolpert, D. M. (2008). Noise in the nervous system. Nature Reviews Neuroscience , 9 (4), 292–303.

Grimm, P. (2010). Social desirability bias. Wiley international encyclopedia of marketing. 

Gundersen, O. E. (2021). The fundamental principles of reproducibility. Philosophical Transactions of the Royal Society A, 379(2197), 20200210.

Harvey Teresa Ewart Robert West, N. (1997). Effects of data noise on statistical judgement. Thinking & Reasoning, 3(2), 111–132.

Hibbert, D. B. (2007). Systematic errors in analytical measurement results. Journal of Chromatography A , 1158 (1–2), 25–32.

Järvinen, T. L., Sihvonen, R., Bhandari, M., Sprague, S., Malmivaara, A., Paavola, M., & Guyatt, G. H. (2014). Blinded interpretation of study results can feasibly and effectively diminish interpretation bias. Journal of Clinical Epidemiology , 67 (7), 769–772.

Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement , 30 (1), 61–70.

McKinnon, P. J. (2013). Maintaining genome stability in the nervous system. Nature Neuroscience , 16 (11), 1523–1529.

McNutt, M. (2014). Reproducibility Science , 343(6168), 229–229.

Nederhof, A. J. (1985). Methods of coping with social desirability bias: A review. European Journal of Social Psychology , 15 (3), 263–280.

Popper, K. (2014). Conjectures and refutations: The growth of scientific knowledge . Routledge.

Scales, J. A., & Snieder, R. (1998). What is noise? Geophysics , 63 (4), 1122–1124.

Sripada, C., Kessler, D., & Jonides, J. (2016). Sifting signal from noise with replication science. Perspectives on Psychological Science , 11 (4), 576–578.

Taleb, N. N. (2016). Fooled by randomness: The hidden role of chance in life and in the markets . Editeurs divers USA.

Thagard, P., & Nisbett, R. E. (1982). Variability and confirmation. Philosophical Studies: An International Journal for Philosophy in the Analytic Tradition , 42 (3), 379–394.

Worrall, J. (2002). What evidence in evidence-based medicine? Philosophy of Science , 69 (S3), S316–S330.

Download references

Acknowledgements

The authors acknowledge the financial support of the University of Graz.

Open access funding provided by University of Graz.

Author information

Authors and affiliations.

Institute of Psychology, University of Graz, Graz, Austria

Michael Vogrin & Jan Willem Koten

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Michael Vogrin .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Vogrin, M., Koten, J.W. RUM leads to noise: the significance of finding the sources of variability between experimental runs. Synthese 204 , 147 (2024). https://doi.org/10.1007/s11229-024-04798-3

Download citation

Received : 23 March 2024

Accepted : 15 October 2024

Published : 04 November 2024

DOI : https://doi.org/10.1007/s11229-024-04798-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Reproducibility
• Variability
• Philosophy of science
• Epistemology
• Find a journal
• Publish with us
• Track your research