As an experimental psychologist, I see many ways in which this QMP approach can be applied to the practice of experimental manipulation - in which participants' psychological states are systematically influenced (or not) by experimenters, based on random assignment. If questionable practices are present in experimental psychology (and there's good reason to think that they are), then there must be Questionable MAnipulation Practices (QMAPs). Adapting Flake and Fried's definition, QMAPs are decisions relating to the conduct of experimental manipulations that lack an empirical justification or transparency. They may or may not undermine the validity or reliability of a given manipulation, but more often than not, such questionable practices are harmful to psychological science. The 'replicability crisis' in experimental psychology may be due, in part, to undisclosed and questionably-justified procedures regarding experimental manipulations (i.e., QMAPs) that yielded false positives and therefore non-replicable findings.
Below, I discuss QMAPs in the selection, creation, modification, and implementation phases of experimental manipulations in psychology.
Selection & Creation
Right off the bat I'm at a crossroads - I can seek out an existing manipulation or create one myself. Without a clear theoretical justification* for one or the other (or without transparently documenting this decision process), I'm already into the tall grass of QMAP territory.
Let's say that I decide to use an existing manipulation because I think it's better to use one with demonstrated efficacy and validity than to go through the fraught process of making one myself. Again, I'm faced with multiple decisions. I can manipulate rejection by:
-leaving participants out of a ball-tossing game
-asking participants to recall an experience of rejection from their past
-telling participants that their personality indicates they'll be alone for the rest of their life
-telling participants that other people chose not to interact with them on a cooperative task
-priming them with images of disapproving faces
-being left out of an online chatroom conversation
-the list goes on and on.....
Which manipulation do I choose? Compounding this problem, a single manipulation is often implemented in many different forms in the published literature. So even after you have chosen a given manipulation, you may have to choose between several different versions of thereof. For instance, do I pick the Cyberball rejection manipulation with 2 other players or 3 other players? Do I use the one that includes face images of the partners or just the virtual avatars? So on and so on ......
As Flake and Fried point out, your true guide in this situation is theory. A solid and nuanced theoretical understanding of the construct you seek to manipulate will allow you to identify features that an experimental manipulation must have (and must not have) to be a valid approach to altering your intended construct. So long as your decision is based on sound theory and transparently disclosed, you can avoid engaging in QMAPs at this stage.
However, there may be no manipulation for your intended construct and you will have to create one yourself! The development of a new experimental manipulation entails innumerable decisions (e.g., online or in-person? simulate the experience with confederates or a computer program? deception or no deception?). The best practices for developing new manipulations is detailed elsewhere and beyond the scope of this piece. My hope is simply to draw awareness to the fact that each of the decisions we make when we develop a new manipulation have the possibility to be QMAPs.
*I fully acknowledge that finding a 'theoretical justification' for a manipulation-relevant decision is not hard to do, especially in the absence of strong theory. Many heuristic-based and intuitive choices can be put forth post hoc as theoretically-justified. This problematic practice has a simple solution --- preregistration.
Although modifications such as cutting trials may seem like a sound idea for logistical reasons, they aren't justified by a solid theoretical rationale. Without this, we don't know if reducing the number of trials affects the underlying ability of the task to manipulate the construct of interest.
Again, we see that there are many ways in which flexibility is built into the implementation phase. It may seem sensible to drop your suspicious participants, but perhaps your manipulation isn't actually invalidated among suspicious participants (or you didn't measure suspicion well) and you're now just removing a group of participants who happen to be lower in, let's say, agreeableness - thus biasing your study's sample and undermining the validity of your findings. Without a good theoretical rationale or empirical evidence, we just can't say whether it's a good approach or not.
-solid and ongoing training in theory, experimental methodology, and construct validation
-detailed and transparent Methods sections
We should commit ourselves to training the next generation of experimentalists in these areas and practicing them ourselves.