Measuring and manipulating psychological processes is the focus of methodological training in psychological research. Relevant to this, two of my favorite scholars to follow on Twitter (Jessica Kay Flake and Eiko Fried) have recently articulated something that has captured my interest in their 2018 APS talk: Questionable Measurement Practices (QMPs). QMPs reflect choices that investigators make when they seek to measure a psychological construct and can undermine the validity of the assessment of psychological constructs. According to their definition, a QMP is "a measurement decision that lacks justification and/or transparency (slide 21)." Flake and Fried then subdivide QMPs into decisions that are made during the Selection of the measure, the Use of the measure, and the Modification of the measure.
As an experimental psychologist, I see many ways in which this QMP approach can be applied to the practice of experimental manipulation - in which participants' psychological states are systematically influenced (or not) by experimenters, based on random assignment. If questionable practices are present in experimental psychology (and there's good reason to think that they are), then there must be Questionable MAnipulation Practices (QMAPs). Adapting Flake and Fried's definition, QMAPs are decisions relating to the conduct of experimental manipulations that lack an empirical justification or transparency. They may or may not undermine the validity or reliability of a given manipulation, but more often than not, such questionable practices are harmful to psychological science. The 'replicability crisis' in experimental psychology may be due, in part, to undisclosed and questionably-justified procedures regarding experimental manipulations (i.e., QMAPs) that yielded false positives and therefore non-replicable findings.
Below, I discuss QMAPs in the selection, creation, modification, and implementation phases of experimental manipulations in psychology.
As an experimental psychologist, I see many ways in which this QMP approach can be applied to the practice of experimental manipulation - in which participants' psychological states are systematically influenced (or not) by experimenters, based on random assignment. If questionable practices are present in experimental psychology (and there's good reason to think that they are), then there must be Questionable MAnipulation Practices (QMAPs). Adapting Flake and Fried's definition, QMAPs are decisions relating to the conduct of experimental manipulations that lack an empirical justification or transparency. They may or may not undermine the validity or reliability of a given manipulation, but more often than not, such questionable practices are harmful to psychological science. The 'replicability crisis' in experimental psychology may be due, in part, to undisclosed and questionably-justified procedures regarding experimental manipulations (i.e., QMAPs) that yielded false positives and therefore non-replicable findings.
Below, I discuss QMAPs in the selection, creation, modification, and implementation phases of experimental manipulations in psychology.
Selection & Creation
Once you have identified a psychological state that you seek to manipulate, you have entered the 'selection' phase. Your goal here is to find or create a valid and reliable manipulation to employ to test your hypothesis. Let's say that I have crafted a hypothesis that entails the experimental manipulation of social rejection.
Right off the bat I'm at a crossroads - I can seek out an existing manipulation or create one myself. Without a clear theoretical justification* for one or the other (or without transparently documenting this decision process), I'm already into the tall grass of QMAP territory.
Let's say that I decide to use an existing manipulation because I think it's better to use one with demonstrated efficacy and validity than to go through the fraught process of making one myself. Again, I'm faced with multiple decisions. I can manipulate rejection by:
-leaving participants out of a ball-tossing game
-asking participants to recall an experience of rejection from their past
-telling participants that their personality indicates they'll be alone for the rest of their life
-telling participants that other people chose not to interact with them on a cooperative task
-priming them with images of disapproving faces
-being left out of an online chatroom conversation
-the list goes on and on.....
Which manipulation do I choose? Compounding this problem, a single manipulation is often implemented in many different forms in the published literature. So even after you have chosen a given manipulation, you may have to choose between several different versions of thereof. For instance, do I pick the Cyberball rejection manipulation with 2 other players or 3 other players? Do I use the one that includes face images of the partners or just the virtual avatars? So on and so on ......
As Flake and Fried point out, your true guide in this situation is theory. A solid and nuanced theoretical understanding of the construct you seek to manipulate will allow you to identify features that an experimental manipulation must have (and must not have) to be a valid approach to altering your intended construct. So long as your decision is based on sound theory and transparently disclosed, you can avoid engaging in QMAPs at this stage.
However, there may be no manipulation for your intended construct and you will have to create one yourself! The development of a new experimental manipulation entails innumerable decisions (e.g., online or in-person? simulate the experience with confederates or a computer program? deception or no deception?). The best practices for developing new manipulations is detailed elsewhere and beyond the scope of this piece. My hope is simply to draw awareness to the fact that each of the decisions we make when we develop a new manipulation have the possibility to be QMAPs.
Note:
*I fully acknowledge that finding a 'theoretical justification' for a manipulation-relevant decision is not hard to do, especially in the absence of strong theory. Many heuristic-based and intuitive choices can be put forth post hoc as theoretically-justified. This problematic practice has a simple solution --- preregistration.
Right off the bat I'm at a crossroads - I can seek out an existing manipulation or create one myself. Without a clear theoretical justification* for one or the other (or without transparently documenting this decision process), I'm already into the tall grass of QMAP territory.
Let's say that I decide to use an existing manipulation because I think it's better to use one with demonstrated efficacy and validity than to go through the fraught process of making one myself. Again, I'm faced with multiple decisions. I can manipulate rejection by:
-leaving participants out of a ball-tossing game
-asking participants to recall an experience of rejection from their past
-telling participants that their personality indicates they'll be alone for the rest of their life
-telling participants that other people chose not to interact with them on a cooperative task
-priming them with images of disapproving faces
-being left out of an online chatroom conversation
-the list goes on and on.....
Which manipulation do I choose? Compounding this problem, a single manipulation is often implemented in many different forms in the published literature. So even after you have chosen a given manipulation, you may have to choose between several different versions of thereof. For instance, do I pick the Cyberball rejection manipulation with 2 other players or 3 other players? Do I use the one that includes face images of the partners or just the virtual avatars? So on and so on ......
As Flake and Fried point out, your true guide in this situation is theory. A solid and nuanced theoretical understanding of the construct you seek to manipulate will allow you to identify features that an experimental manipulation must have (and must not have) to be a valid approach to altering your intended construct. So long as your decision is based on sound theory and transparently disclosed, you can avoid engaging in QMAPs at this stage.
However, there may be no manipulation for your intended construct and you will have to create one yourself! The development of a new experimental manipulation entails innumerable decisions (e.g., online or in-person? simulate the experience with confederates or a computer program? deception or no deception?). The best practices for developing new manipulations is detailed elsewhere and beyond the scope of this piece. My hope is simply to draw awareness to the fact that each of the decisions we make when we develop a new manipulation have the possibility to be QMAPs.
Note:
*I fully acknowledge that finding a 'theoretical justification' for a manipulation-relevant decision is not hard to do, especially in the absence of strong theory. Many heuristic-based and intuitive choices can be put forth post hoc as theoretically-justified. This problematic practice has a simple solution --- preregistration.
Modification
Once you've chosen a manipulation, you must then consider whether to implement it exactly as it was detailed in the literature in which you found it, or to modify it in some way. Any deviation from the published description can become a QMAP if not justified and openly-disclosed. Below, I list some hypothetical examples of how modifications to a published experimental manipulation can be made and how they may be justified in ways that stem from practical or other considerations, instead of theory.
Although modifications such as cutting trials may seem like a sound idea for logistical reasons, they aren't justified by a solid theoretical rationale. Without this, we don't know if reducing the number of trials affects the underlying ability of the task to manipulate the construct of interest.
Implementation
Now that you have either obtained or created a suitable manipulation and modified it to your satisfaction, you have to look to its implementation. This is one of the least talked-about and most difficult phases of an experiment - yet if done improperly, it can entirely undermine the validity of your study. Below, I list some hypothetical examples of choices investigators often make in this implementation phase and ways in which they could be justified without theory.
Again, we see that there are many ways in which flexibility is built into the implementation phase. It may seem sensible to drop your suspicious participants, but perhaps your manipulation isn't actually invalidated among suspicious participants (or you didn't measure suspicion well) and you're now just removing a group of participants who happen to be lower in, let's say, agreeableness - thus biasing your study's sample and undermining the validity of your findings. Without a good theoretical rationale or empirical evidence, we just can't say whether it's a good approach or not.
Conclusion
Experimentally-manipulating psychological constructs is a crucial means to establish causal effects. The validity of such manipulations are crucial to the credibility of experimental psychology. QMAPs have the potential to undercut this validity. At the end of the day, the best weapons we have against QMAPs are:
-solid and ongoing training in theory, experimental methodology, and construct validation
-open science
-detailed and transparent Methods sections
-preregistration
We should commit ourselves to training the next generation of experimentalists in these areas and practicing them ourselves.
-solid and ongoing training in theory, experimental methodology, and construct validation
-open science
-detailed and transparent Methods sections
-preregistration
We should commit ourselves to training the next generation of experimentalists in these areas and practicing them ourselves.