The final version of the Measurement Schmeasurement paper was released by JK Flake and Eiko Fried, which expertly highlights psychology's measurement crisis surrounding Questionable Measurement Practices (QMPs). I get the vibe that folks have largely been persuaded that this crisis is real, QMPs exist and should be eliminated, and the psychometric practices of our field need an overhaul. At the same time, I've been trying to ring the bell about similar shortcomings in our field's Questionable MAnipulation Practices (QMAPS) with much less success. I'm not sure if that's just due to my own failings as a science advocate or if folks don't think there's a big problem with how we're approaching the manipulation of psychological variables. For anyone who remains unconvinced, there's a new paper in Perspectives on Psychological Science provides more evidence for just how bad things might be.
This paper critiqued the power posing literature. Power posing studies manipulate powerful postures by asking participants to either adopt an expansive body stance (see below) or a control stance. In the control condition, participants adopt a contractive stance (e.g., hugging yourself, head down, legs pulled together).
This paper critiqued the power posing literature. Power posing studies manipulate powerful postures by asking participants to either adopt an expansive body stance (see below) or a control stance. In the control condition, participants adopt a contractive stance (e.g., hugging yourself, head down, legs pulled together).
These manipulations are then used to show that such 'power poses' make people feel and behave more assertively, inter alia. But the use of an active control condition (i.e., a control condition where the participant completes a task that is intended to induce a dissimilar psychological state from the experimental condition) prevents us from knowing what participants would have done and felt in the absence of instructions to strike a pose. As such, we don't know which of the two conditions is driving the manipulation's effect. The relatively greater assertiveness observed among those who strike an expansive (versus contractive) pose is often interpreted as being attributable to the expansive posture condition (i.e., the power pose), but it's just as likely that the contractive posture (i.e., the active control condition) leads people to be less assertive!
There are 3 possible configurations for how a promising difference between an experimental condition and active control condition might arise. In Scenario A (see below), both the experimental and active control conditions exert their intended effects. Specifically, participants in the experimental condition (e.g., expansive posture) show a +0.5 increase from baseline (i.e., the passive control condition) and participants in the active control condition (e.g., constrictive posture) show a -0.5 decrease from baseline, for a mean difference of 1.0 between the experimental and active control conditions.
In Scenario B (see below), only the experimental condition exerts its intended effect. Specifically, participants in the experimental condition (e.g., expansive posture) show a +1.0 increase from baseline (i.e., the passive control condition) and participants in the active control condition (e.g., constrictive posture) show no change from baseline, for a mean difference of 1.0 between the experimental and active control conditions.
In Scenario C (see below), only the active control condition exerts its intended effect. Specifically, participants in the experimental condition (e.g., expansive posture) show no change from baseline (i.e., the passive control condition) and participants in the active control condition (e.g., constrictive posture) show a -1.0 decrease from baseline, for a mean difference of 1.0 between the experimental and active control conditions.
All 3 scenarios produce the same observed difference between experimental and control conditions, but from meaningfully different patterns of effects. Scenarios A and B are what most investigators want, as the experimental condition is having the intended effect. Yet without a 'neutral', passive control condition that captures participants' baseline, experimenters cannot know if their effect is actually a reflection of Scenario C.
The new meta-analysis sought to examine which of these 3 scenarios was most likely present in the power posing literature. The authors did so by examining power posing studies that also included 'neutral' conditions, in which participants were not instructed to strike either an expansive or contractive posture. The meta-analysis compared expansive and contractive posture conditions against this neutral condition to test which of these two conditions was doing the heavy-lifting. Counter to claims from the power posing literature, the meta-analysis found that comparing the expansive posture condition to the neutral condition returned a null overall effect, g = .06, p = .197. The meta-analysis went on to find that comparing the contractive control condition to the neutral condition returned a large overall effect, g = .45, p < .001. These findings suggest that the difference between the experimental and active control conditions of power pose manipulations was due to effects from the active control condition and not the experimental condition (i.e., Scenario C)!
The main take home from this paper is best stated by the authors: "...the results point to the importance of including a neutral control condition in experimental studies of the effect of manipulating motor displays, allowing the effect to be ascribed to the appropriate condition." Manipulations refer to specific assemblages of experimental conditions and a manipulation cannot be said to be validated unless it has been tested alongside both passive and active control conditions and the results reflect Scenarios A or B.
So what should a passive control condition look like? This will be a unique consideration for each manipulation, but the best advice may be just to always include a third condition in which participants skip as much of the manipulation procedures as possible. For instance, Cyberball is a widely-used social exclusion manipulation that most frequently compares an experimental exclusion condition (where participants are left out of a ball-toss) with an active inclusion control condition (where participants receive an equal number of ball-tosses). A passive control condition might just have participants skip the manipulation task completely and immediately begin on whatever procedures follow the ball-tossing task.
Skipping such manipulation procedures may not always be possible (e.g., the Cyberball task is a necessary setup for the subsequent task). In such cases, it's likely best to make the passive condition as 'neutral' as possible. More specifically, experimenters should seek to make the passive control condition replicate the routine, mundane, expected, and run-of-the-mill experiences of everyday life. Doing so allows participants to engage in the psychological processes that are independent of the constraints and demands of experimental manipulations, thus approximating a baseline comparator for the other conditions.
The new meta-analysis sought to examine which of these 3 scenarios was most likely present in the power posing literature. The authors did so by examining power posing studies that also included 'neutral' conditions, in which participants were not instructed to strike either an expansive or contractive posture. The meta-analysis compared expansive and contractive posture conditions against this neutral condition to test which of these two conditions was doing the heavy-lifting. Counter to claims from the power posing literature, the meta-analysis found that comparing the expansive posture condition to the neutral condition returned a null overall effect, g = .06, p = .197. The meta-analysis went on to find that comparing the contractive control condition to the neutral condition returned a large overall effect, g = .45, p < .001. These findings suggest that the difference between the experimental and active control conditions of power pose manipulations was due to effects from the active control condition and not the experimental condition (i.e., Scenario C)!
The main take home from this paper is best stated by the authors: "...the results point to the importance of including a neutral control condition in experimental studies of the effect of manipulating motor displays, allowing the effect to be ascribed to the appropriate condition." Manipulations refer to specific assemblages of experimental conditions and a manipulation cannot be said to be validated unless it has been tested alongside both passive and active control conditions and the results reflect Scenarios A or B.
So what should a passive control condition look like? This will be a unique consideration for each manipulation, but the best advice may be just to always include a third condition in which participants skip as much of the manipulation procedures as possible. For instance, Cyberball is a widely-used social exclusion manipulation that most frequently compares an experimental exclusion condition (where participants are left out of a ball-toss) with an active inclusion control condition (where participants receive an equal number of ball-tosses). A passive control condition might just have participants skip the manipulation task completely and immediately begin on whatever procedures follow the ball-tossing task.
Skipping such manipulation procedures may not always be possible (e.g., the Cyberball task is a necessary setup for the subsequent task). In such cases, it's likely best to make the passive condition as 'neutral' as possible. More specifically, experimenters should seek to make the passive control condition replicate the routine, mundane, expected, and run-of-the-mill experiences of everyday life. Doing so allows participants to engage in the psychological processes that are independent of the constraints and demands of experimental manipulations, thus approximating a baseline comparator for the other conditions.