Uri Simonsohn uncovers a shocking case of fake data in the field of psychology. In this intriguing article, Simonsohn recounts the fortuitous discovery of anomalies in a study involving coin sizes, leading to the retraction of the paper. He highlights implausible correlations among emotion measures and impossibly similar results as key red flags. Simonsohn emphasizes the need for data posting by journals, which would facilitate the identification of invalid data. Dive into this gripping investigation of academic integrity and the quest for truth in scientific research.

The fortuitous discovery of new fake data. For a project I worked on this past May, I needed data for variables as different from each other as possible. From the data-posting journal Judgment and Decision Making I downloaded data for ten, including one from a now retracted paper involving the estimation of coin sizes. I created a chart and inserted it into a paper that I sent to several colleagues, and into slides presented at an APS talk.An anonymous colleague, “Larry,” saw the chart and, for not-entirely obvious reasons, became interested in the coin-size study. After downloading the publicly available data he noticed something odd (something I had not noticed): while each participant had evaluated four coins, the data contained only one column of estimates. The average? No, for all entries were integers; averages of four numbers are rarely integers. Something was off.Interest piqued, he did more analyses leading to more anomalies. He shared them with the editor, who contacted the author. The author provided explanations. These were nearly as implausible as they were incapable of accounting for the anomalies. The retraction ensued.Some of the anomalies 1. Contradiction with paper Paper describes 0-10 integer scale, dataset has decimals and negative numbers. 2. Implausible correlations among emotion measures Shame and embarrassment are intimately related emotions, and yet they are correlated negatively in the data r = -.27. Fear and anxiety: r = -.01. Real emotion ratings don’t exhibit these correlations.3. Impossibly similar results Fabricated data often exhibit a pattern of excessive similarity (e.g., very similar means across conditions). This pattern led to uncovering Sanna and Smeesters as fabricateurs (see “Just Post It” paper). Diederik Stapel’s data also exhibit excessive similarity, going back to his dissertation at least.The coin-size paper also has excessive similarity. For example, coin-size estimates supposedly obtained from 49 individuals across two different experiments are almost identical: Experiment 1 (n=25): 2,3,3,3,3,4,4,4,4,4,5,5,5,5,5,5,5,6,6,6,6,6,6,6,7 Experiment 2 (n=24): 2,3,3,3,3,4,4,4,4,4,5,5,5,5,5,5,_,6,6,6,6,6,6,6,7Simulations drawing random samples from the data themselves (bootstrapping) show that it is nearly impossible to obtain such similar results. The hypothesis that these data came from random samples is rejected, p<.000025 (see R code, detailed explanation). Who vs. which These data are fake beyond reasonable doubt.  We don’t know, however, who faked them. That question is of obvious importance to the authors of the paper and perhaps their home and granting institutions, but arguably not so much to the  research community more broadly. We should care, instead, about which data are fake.If other journals followed the lead of Judgment and Decision Making and required data posting (its  editor Jon Baron, by the way,  started the data posting policy well before I wrote my “Just Post It”), we would have a much easier time identifying invalid data.  Some of the coin-size authors have  a paper in JESP, one in Psychological Science, and another with similar results  in Appetite.  If the data behind those papers were available, we would not need to speculate as to their validity.Author's response When discussing the work of others, our policy here at Data Colada is to contact them before posting. We ask for feedback to avoid inaccuracies and misunderstandings, and  give authors space for commenting within our original blog post. The corresponding author of the retracted article,  Dr. Wen-Bin Chiou, wrote to me via email:Although the data collection and data coding was done by my research assistant, I must be responsible for the issue.Unfortunately, the RA had left my lab last year and studied abroad. At this time, I cannot get the truth from him and find out what was really going wrong […] as to the decimal points and negative numbers, I recoded the data myself and sent the editor with the new dataset. I guess the problem does not exist in the new dataset. With regard to the impossible similar results, the RA sorted the coin-size estimate variable, producing the similar results. […]  Finally, I would like to thank Dr. Simonsohn for including my clarifications in this post. [See unedited version] Uri's note: the similarity of data is based on the frequency of values across samples, not their order, so sorting does not explain  that the data are incompatible with random sampling.

[1] "Just Posting It" works, leads to new retraction in Psychology

Leif Nelson takes a lighthearted approach in this article as he discusses how he uses his personal listening habits to test his students' ability to predict his favorite songs. Through analyzing his iTunes data, Nelson explores the statistical realities and nuances of constructing a good mixtape. From the impact of song duration to the influence of personal preferences, this article offers a witty and insightful perspective on the intersection of music and data analysis. Don't miss out on this entertaining read that may even inspire you to question your own listening habits.

Not everything at Data Colada is as serious as fraudulent data. This post is way less serious than that. This post is about music and teaching.As part of their final exam, my students analyze a data set. For a few years that data set has been a collection of my personal listening data from iTunes over the previous year. The data set has about 500 rows, with each reporting a song from that year, when I purchased it, how many times I listened to it, and a handful of other pieces of information. The students predict the songs I will include on my end-of-year “Leif’s Favorite Songs” compact disc. (Note to the youth: compact discs were physical objects that look a lot like Blu-Ray discs. We used to put them in machines to hear music.) So the students are meant to combine regressions and intuitions to make predictions. I grade them based on how many songs they correctly predict. I love this assignment.The downside, as my TA tells me, is that my answer key is terrible. The problem is that I am encumbered both by my (slightly) superior statistical sense and my (substantially) superior sense of my own intentions and preferences. You see, a lot goes into the construction of a good mix tape (Note to the youth: tapes were like CD’s, except if you wanted to hear track 1 and then track 8 you were SOL.) I expected my students to account for that. “Ah look,” I am picturing, “he listened a lot to Pumped Up Kicks. But that would be an embarrassing pick. On the other hand, he skipped this Gil Scott-Heron remix a lot, but you know that’s going on there.” They don’t do that. They pick the songs I listen to a lot.But then they miss certain statistical realities. When it comes to grading, the single biggest differentiator is whether or not a student accounts for how long a song is in the playlist (see the scatterplot of 2011, below). If you don’t account for it, then you think that all of my favorite songs were released in the first couple of months. A solid 50% of students think that I have a mad crush on January music. The other half try to account for it. Some calculate a “listens per day” metric, while others use a standardization procedure of one type or another. I personally use a method that essentially accounts for the likelihood that a song will come up, and therefore heavily discounts the very early tracks and weighs the later tracks all about the same. You may ask, “wait, why are you analyzing your own data?” No good explanation. I will say though, I almost certainly change my preferences based on these analyses – I change them away from what my algorithm predicts. That is bad for the assignment. I am not a perfect teacher.I don’t think that I will use this assignment anymore since I no longer listen to iTunes. Now I use Spotify. (Note to the old: Spotify is like a musical science fiction miracle that you will never understand. I don’t.)

[2] Using Personal Listening Habits to Identify Personal Music Preferences

In this intriguing article, Joe Simmons investigates a fascinating phenomenon called the Unit Asking Effect. He delves into a new study that explores how people are more inclined to donate more money to help a group of people if they are first asked how much they would donate to help just one person. Simmons replicates the study and shares his findings, highlighting some interesting differences from the original research. Discover the power of psychology in designing effective interventions and the potential impact it can have on charitable donations.

A new paper finds that people will donate more money to help 20 people if you first ask them how much they would donate to help 1 person.This Unit Asking Effect (Hsee, Zhang, Lu, & Xu, 2013, Psychological Science) emerges because donors are naturally insensitive to the number of individuals needing help. For example, Hsee et al. observed that if you ask different people how much they’d donate to help either 1 needy child or 20 needy children, you get virtually the same answer. But if you ask the same people to indicate how much they’d donate to 1 child and then to 20 children, they realize that they should donate more to help 20 than to help 1, and so they increase their donations.If true, then this is a great example of how one can use psychology to design effective interventions.The paper reports two field experiments and a study that solicited hypothetical donations (Study 1). Because it was easy, I attempted to replicate the latter. (Here at Data Colada, we report all of our replication attempts, no matter the outcome).I ran two replications, a “near replication” using materials that I developed based on the authors’ description of their methods (minus a picture of a needy schoolchild) and then an “exact replication” using the authors’ exact materials. (Thanks to Chris Hsee and Jiao Zhang for providing those).In the original study, people were asked how much they’d donate to help a kindergarten principal buy Christmas gifts for her 20 low-income pupils. There were four conditions, but I only ran the three most interesting conditions: The original study had ~45 participants per cell. To be properly powered, replications should have ~2.5 times the original sample size. I (foolishly) collected only ~100 per cell in my near replication, but corrected my mistake in the exact replication (~150 per cell). Following Hsee et al., I dropped responses more than 3 SD from the mean, though there was a complication in the exact replication that required a judgment call. My studies used MTurk participants; theirs used participants from “a nationwide online survey service.”Here are the results of the original (some means and SEs are guesses) and my replications (full data).I successfully replicated the Unit Asking Effect, as defined by Unit Asking vs. Control; it was marginal (p=.089) in the smaller-sampled near replication and highly significant (p< .001) in the exact replication.There were some differences. First, my effect sizes (d=.24 and d=.48) were smaller than theirs (d=.88). Second, whereas they found that, across conditions, people were insensitive to whether they were asked to donate to 1 child or 20 children (the white $15 bar vs. the gray $18 bar), I found a large difference in my near replication and a smaller but significant difference in the exact replication. This sensitivity is important, because if people do give lower donations for 1 child than for 20, then they might anchor on those lower amounts, which could diminish the Unit Asking Effect.In sum, my studies replicated the Unit Asking Effect.

[3] A New Way To Increase Charitable Donations: Does It Replicate?

Uri Simonsohn challenges the common practice of powering replications based solely on observed effect size. In this thought-provoking essay, he sheds light on the misleading nature of this approach due to publication bias. Simonsohn argues that overestimating effect sizes in original research leads to overestimating the power of subsequent replications. He proposes an alternative approach to evaluating replication results, focusing on tight confidence intervals around zero. Discover why Simonsohn's new perspective on sample size and statistical power brings fresh insights to the field of research.

It is common for researchers running replications to set their sample size assuming the effect size the original researchers got is correct. So if the original study found an effect-size of d=.73, the replicator assumes the true effect is d=.73, and sets sample size so as to have 90% chance, say, of getting a significant result.This apparently sensible way to power replications is actually deeply misleading.Why Misleading? Because of publication bias. Given that (original) research is only publishable if it is significant, published research systematically overestimates effect size (Lane & Dunlap, 1978). For example, if sample size is n=20 per cell, and true effect size is d=.2, published studies will on average estimate the effect to be d=.78. The intuition is that overestimates are more likely to be significant than underestimates, and so more likely to be published.If we systematically overestimate effect sizes in original work, then we systematically overestimate the power of replications that assume those effects are real.Let’s consider some scenarios. If original research were powered to 50%, a highly optimistic benchmark (Button et al, 2013; Sedlmeier & Gigerenzer, 1989), here is what it looks like: So replications claiming 80% power actually have just 51% (Details | R code).Ok. What if original research were powered at a more realistic level of, say, 35%:  The figures show that the extent of overclaiming depends on the power of the original study. Because nobody knows what that is, nobody knows how much power a replication claiming 80%, 90% or 95% really has.A self-righteous counterargument A replicator may say:Well, if the original author underpowered her studies, then she is getting what she deserves when the replications fail; it is not my fault my replication is underpowered, it is hers. SHE SHOULD BE DOING POWER ANALYSIS!!!Three problems. 1. Replications in particular and research in general are not about justice. We should strive to maximize learning, not schadenfreude.2. The original researcher may have thought the effect was bigger than it is, she thought she had  80% power, but she had only 50%. It is not “fair” to "punish" her for not knowing the effect size she is studying. That's precisely why she is studying it.3. Even if all original studies had 80% power, most published estimates would be over-estimates, and so even if  all original studies had 80% power, most replications based on observed effects would overclaim power. For instance, one in five replications claiming 80% would actually have <50% power (R code). What’s the alternative? In a recent paper (“Evaluating Replication Results”) I put forward a different approach to thinking about replication results altogether. For a replication to fail it is not enough that p>.05 in it, we need to also conclude the effect is too small to have been detected in the original study (in effect, we need tight confidence intervals around 0). Underpowered replications will tend to fail to reject 0, be n.s., but will also tend to fail to reject big effects. In the new approach this result is considered as uninformative rather than as a "failure-to-replicate." The paper also derives a simple rule for sample size to be properly powered for obtaining informative failures to replicate:  2.5 times the original sample size ensures 80% power for that test. That number is unaffected by publication bias, how original authors power their studies, and the study design (e.g., two-proportions vs. ANOVA).Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe

[4] The Folly of Powering Replications Based on Observed Effect Size

Leif Nelson delves into the fascinating world of random numbers and explores the surprising patterns and preferences that emerge from our supposedly random choices. By comparing favorite numbers to numbers generated at random, Nelson reveals intriguing correlations and sheds light on our biases when it comes to number selection. From debunking the notion of truly random choices to uncovering the numerical trends in PIN codes, this thought-provoking article challenges our understanding of randomness and offers insight into the human mind's relationship with numbers. Prepare to question everything you thought you knew about randomness!

What’s your favorite number between 1 and 100? Now, think of a random number between 1 and 100. My goal for this post is to compare those two responses.Number preferences feel random. They aren’t. “Random” numbers also feel random. Those aren’t random either. I collected some data, found a pair of austere academic papers, and one outstanding blog post. I will tell you about all of them.First, the data I collected. I (along with Hannah Perfecto, one of my excellent doctoral students) asked one group of people to generate a random number between 1 and 100. Another group reported their favorite number between 1 and 100. That’s it.We know a little about preferences. People like their birthday numbers, for example. They pursue round numbers. In preparing this post, I learned of a simmering literature on single-digit number preferences, suggesting that in both 1971 and in 1988 people liked the number 7. (Aside: Someone should write the number preference equivalent of the Princeton Trilogy. In fact, why not move beyond preferences to other attributes? For example, are even numbers more warm or more competent?*). As far as I can tell, less is known about how people generate random numbers. Do people choose the same numbers at random as they choose as their favorites?The figures tell the whole story, but words are useful. Consider four notable numbers. Consistent with past research, people like the number 7. Inconsistent with horror movie titlers and hotel floor number assigners, people also like the number 13. The number 42 has an entirely wonderful Wikipedia entry, suggesting that its consequence goes beyond Jackie Robinson and Douglas Adams. Perhaps the Data Colada can add a small footnote to its mystique? Finally, the number 69 also has a Wikipedia entry, though it is far less vivid than you’re anticipating. On the random side there are fewer obvious winners (three way tie between 5, 67, and 69). How about some other patterns? First of all, the two sets are highly, but imperfectly, correlated at r = .48. Random numbers are larger than favorite numbers (Ms = 46.9 vs. 30.7), t(565) = 7.01, pThese tendencies are partially reflected in the numeric codes people choose for debit cards and their ilk. PIN numbers are a mix of preference and random, and consistent with the data we collected, a brilliant analysis of leaked PIN numbers reveals birthday liking (numbers below 32) and repeated numbers (like multiples of 11). Figure 3 reproduces a chart of 4-digit PIN codes. It will take 30 seconds to orient yourself, but then you will spend five minutes savoring it. My favorite number is just about the most arbitrary preference possible. My “random” number is more arbitrary. But neither is arbitrary at all.* Hypothesis: More warm. Odd numbers are wicked competent.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe

[5] The Consistency of Random Numbers

Joe Simmons challenges the criticism of studies with "overpowered" sample sizes in this thought-provoking essay. He argues that larger sample sizes do not change effect sizes and are beneficial in increasing the precision of results. Simmons emphasizes the problem of underpowered studies in the field and criticizes the nonsensical argument against large samples. As an empirical science, Simmons emphasizes the importance of collecting more data and encourages readers to rethink the methodological norms. Discover why larger sample sizes should never be criticized in this enlightening piece.

Reviewers, and even associate editors, sometimes criticize studies for being “overpowered” – that is, for having sample sizes that are too large. (Recently, the between-subjects sample sizes under attack were about 50-60 per cell, just a little larger than you need to have an 80% chance to detect that men weigh more than women).This criticism never makes sense.The rationale for it is something like this: “With such large sample sizes, even trivial effect sizes will be significant. Thus, the effect must be trivial (and we don’t care about trivial effect sizes).”But if this is the rationale, then the criticism is ultimately targeting the effect size rather than the sample size.  A person concerned that an effect “might” be trivial because it is significant with a large sample can simply compute the effect size, and then judge whether it is trivial.(As an aside: Assume you want an 80% chance to detect a between-subjects effect. You need about 6,000 per cell for a "trivial" effect, say d=.05, and still about 250 per cell for a meaningful "small" effect, say d=.25. We don't need to worry that studies with 60 per cell will make trivial effects be significant).It is OK to criticize a study for having a small effect size. But it is not OK to criticize a study for having a large sample size. This is because sample sizes do not change effect sizes. If I were to study the effect of gender on weight with 40 people or with 400 people, I would, on average, estimate the same effect size (d ~= .59). Collecting 360 additional observations does not decrease my effect size (though, happily, it does increase the precision of my effect size estimate, and that increased precision better enables me to tell whether an effect size is in fact trivial).Our field suffers from a problem of underpowering. When we underpower our studies, we either suffer the consequences of a large file drawer of failed studies (bad for us) or we are motivated to p-hack in order to find something to be significant (bad for the field). Those who criticize studies for being overpowered are using a nonsensical argument to reinforce exactly the wrong methodological norms.If someone wants to criticize trivial effect sizes, they can compute them and, if they are trivial, criticize them. But they should never criticize samples for being too large.We are an empirical science. We collect data, and use those data to learn about the world. For an empirical science, large samples are good. It is never worse to have more data.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe

[6] Samples Can't Be Too Large

In this highly anticipated forthcoming article in the American Economic Review, Uri, Joe, and Leif delve into a misdiagnosed failure-to-replicate a classic study on anchoring effects. The original study by Ariely, Loewenstein, and Prelec showed that high anchors led to higher payments compared to low anchors. However, the AER replication only focused on one study, and the results were consistent with the original but mistakenly interpreted as a failure. Discover why the replication was imprecise, the insights gained from successful replications, and the implications for anchoring effects. Dive into this thought-provoking analysis and challenge your understanding of replication in economics.

In the paper “One Swallow Doesn't Make A Summer: New Evidence on Anchoring Effects”, forthcoming in the AER, Maniadis, Tufano and List attempted to replicate a classic study in economics. The results were entirely consistent with the original and yet they interpreted them as a “failure to replicate.” What went wrong?This post answers that question succinctly; our new paper has additional analyses.Original results In an article with >600 citations, Ariely, Loewenstein, and Prelec (2003) showed that people presented with high anchors (“Would you pay $70 for a box of chocolates?”) end up paying more than people presented with low anchors (“Would you pay $20 for a box of chocolates?”). They found this effect in five studies, but the AER replication reran only Study 2. In that study, participants gave their asking prices for aversive sounds that were 10, 30, or 60 seconds long, after a high (50¢), low (10¢), or no anchor.Replication resultscomparing only the 10-cent and 50-cent anchor conditions, we find an effect size equal to 28.57 percent [the percentage difference between valuations], about half of what ALP found. The p-value […] was equal to 0.253” (p. 8).So their evidence is unable to rule out the possibility that anchoring is a zero effect. But that is only part of the story. Does their evidence also rule out a sizable anchoring effect? It does not. Their evidence is consistent with an effect much larger than the original.Those calculations use Maniadis et al.'s definition of effect size: % difference in valuations (as quoted above). An alternative is to divide the differences of means by the standard deviation (Cohen's d). Using this metric the Replication’s effect size is more markedly different from the Original’s, d=.94 vs. d=.26 . However, the 95% confidence interval for the Replication includes effects as big as d=.64, midway between medium and large effects. Whether we examine Maniadis et al.'s operationalization of effect size, then, or Cohen’s d, we arrive at the same conclusion: the Replication is too noisy to distinguish between a nonexistent and a sizable anchoring effect.Why is the Replication so imprecise? In addition to having 12% fewer participants, nearly half of all valuations are ≤10¢. Even if anchoring had a large percentage effect, one that doubles WTA from 3¢ to 6¢, the tendency of participants to round both to 5¢ makes it undetectable. And there is the floor effect: valuations so close to $0 cannot drop. One way around this problem is to do something economists do all the time: Express the effect size of one variable (How big is the impact of X on Z?) relative to the effect size of another (it is half the effect of Y on Z). Figure 2 shows that, in cents, both the effect of anchoring and duration is smaller in the replication, and that the relative effect of anchoring is comparable across studies. p-curve The original paper had five studies, four were p<.01, the fifth p<.02. When we submit these p-values to p-curve we can empirically examine the fear expressed by the replicators that the original finding is false-positive. The results strongly reject this possibility; selective reporting is an unlikely explanation for the original paper, p<.0001.Some successful replications Every year Uri runs a replication of Ariely et al.'s Study 1 in his class. In an online survey at the beginning of the semester, students write down the last two digits of their social-security-number, indicate if they would pay that amount for something (this semester it was for a ticket to watch Jerry Seinfeld live on campus), and then indicate the most they would pay. Figure 3 has this year’s data:We recently learned that SangSuk Yoon, Nathan Fong and Angelika Dimoka successfully replicated Ariely et al.'s Study 1 with real decisions (in contrast to this paper).Concluding remark We are not vouching for the universal replicability of Ariely et al here. It is not difficult to imagine moderators (beyond floor effects) that attenuate anchoring. We are arguing that the forthcoming "failure-to-replicate" anchoring in the AER is no such thing.note: When we discuss others’ work at DataColada we ask them for feedback and offer them space to comment within the original post. Maniadis, Tufano, and List provided feedback only for our paper and did not send us comments to post here.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe

[7] Forthcoming in the American Economic Review: A Misdiagnosed Failure-to-Replicate

Leif Nelson takes us on an intriguing journey in the realm of animal assessments, exploring the surprising correlations between speed and morality. Delving into the intriguing results of a study that asked participants to rate animals based on their speed and moral worth, Nelson uncovers unexpected insights, including the moral superiority of the Tortoise and the peculiar favoritism towards the Jellyfish. With expert opinions and thought-provoking observations, this article challenges our assumptions and offers a fresh perspective on the intricate connections between humans and the animal kingdom. Prepare to be captivated by this unconventional exploration of speed and morality.

In surveys, most people answer most questions. That is true regardless of whether or not questions are coherently constructed and reasonably articulated. That means that absurd questions still receive answers, and in part because humans are similar to one another, those answers can even look peculiarly consistent. I asked an absurd question and was rewarded with an entertaining answer.Some years ago, with Tom Meyvis, I tried to develop a manipulation to create an association between speed and virtue. Our spartan publication history on the topic testifies to our (lack of) success. That doesn’t mean that the pilot data weren’t interesting for a different reason.Participants saw a sequence of 20 animal photographs and rated each on one of two bipolar dimensions: speed or goodness. The former is straightforward. The latter could be best construed as an evaluation of moral worth. That is an absurd question. What sorts of answers did we receive?  My Top 5 observations:1. The Tortoise is the most moral animal. I anticipated more class-profiling, and a resulting ingroup bias for mammalia. Nope. Perhaps researchers should try an implicit measure?*2. Aquatic race featuring: Jellyfish vs. Starfish vs. Walrus. Who wins? People give the jellyfish the edge. The starfish has no chance.3. Nature documentaries frequently bandy about facts like, “hippopotami kill more people every year than heart disease.” My respondents overlooked that; Hippos are more moral than sloths (which nature documentaries never mention for their killing ability).4. The orangutan is not just a mammal or just a primate, it is a great ape. Huge opportunity for some ingroup favoritism. Instead people favor the cheetah, walrus, and hippo (amongst others). Explain that.5. Most animals are good. Our scale had a meaningful midpoint, yet all but three animals are above it. Who is bad? Hyena, Barracuda, and Jellyfish. The Jellyfish is worst. And deceptively fast. Perhaps a researcher could prime people with jellyfish and see if they cheat more on that matrices task?**Perhaps some absurd questions have correct answers? I asked a pair of experts. Pieter Thomas Jefferson Johnson is an ecologist possibly best known for solving a major scientific problem before he was old enough to drink. Michael Jennions is a world renowned evolutionary biologist, known for many things, including this video (the link alone makes this post worthwhile). I asked them to rank the 20 animals for speed and morality. Their speed ratings are similar to each other (r = .91) and the novices (r = .87). Morality was trickier. Both said that any response would be random, or as Piet said, “I would probably tie them all in ranking”. But responses aren’t quite random. Michael rated based on the complexity of the central nervous system (complex = evil), whereas Pieter used “trophic level, followed by an inverse body mass index”. Despite very different approaches, they are mildly correlated with each other (r = .29). Experts and novices all agree on the virtue of the Tortoise, but Michael and Piet are just as fond of the lowly snail.  *No they shouldn’t.**Don’t run that study. I mean it.

[8] Adventures in the Assessment of Animal Speed and Morality

In this quirky and data-driven essay, Uri Simonsohn delves into the fascinating world of titles and their impact. From clichés to word choices, he unveils intriguing facts about the psychology of naming articles. Discover the most frequently used idioms, the rise of colons, and even the highest scoring Scrabble words found in psychology titles. With a touch of humor and some surprising insights, Simonsohn proves that there's more to a title than meets the eye. Get ready for a delightful exploration of the art of naming in academia.

Naming things is fun. Not sure why, but it is. I have collaborated in the naming of people, cats, papers, a blog, its posts, and in coining the term "p-hacking." All were fun to do. So I thought I would write a Colada on titles.To add color I collected some data. At the end what I wrote was quite boring, so I killed it, but the facts seemed worth sharing. Here they go, in mostly non-contextualized prose.Cliché titles I dislike titles with (unmodified) idioms. The figure below shows how frequent some of them are in the web-of-science archive.  Ironically, the most popular (I found), at 970 papers, is “What’s in a name?” …Lack of originality?Colonization A colleague once shared his disapproval of the increase in the use of colons in titles. With this post as an excuse, I used Mozenda to scrape ~30,000 psychology paper titles published across 19 journals over 40 years, and computed the fraction including a colon. “Colleague was Wrong: Title Colonization Has Been Stable at about 63% Since the 1970s.” [1]That factoid took a couple of hours to generate. Data in hand I figured I should answer more questions. Any sense of coherence in this piece disappears with the next pixel.Have titles gotten longer over time?   Yes. At about 1.5 characters per year (or a tweet a century). note: controlling for journal fixed effects.Three less obvious questions to ask Question 1. What are the two highest scoring Scrabble words used in a Psychology title?  Hypnotizability (37 points), is used in several articles it turns out. [2]
Ventriloquized (36 points) only in this paper.Question 2. What is the most frequent last-word in a Psychology paper title? (try guessing before reading the next line)This is probably the right place to let you know the Colada has a Facebook page now Winner: 137 titles end with: “Tasks” Runner up: 70 titles end with “Effect”Question 3. What’s more commonly used in a Psychology title, “thinking” or “sex”? Not close.Sex: 407. Thinking: 172.Alright, that's not totally fair, in psychology sex often refers to gender rather than the activity. Moreover, thinking (172) is, as expected for academic papers, more common than doing (44). But memory blows sex, thinking, and doing combined out of the water with 2008 instances; one in 15 psychology titles has the word memory in them.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe  I treated the Journal of Consumer Research as a psychology journal, a decision involving two debatable assumptions. [↩]Shane Frederick indicated via email that this is a vast underestimate that ignores tripling of points; Hypnotizability could get you 729 points . [↩]

[9] Titleogy: Some facts about titles

Uri, Joe, and Leif discuss the growing trend of requiring authors to disclose flexibility in data collection and analysis in scientific journals. They highlight how these policies align with their own recommendations for increased transparency in research. The authors also propose a standardized disclosure request that reviewers can use when evaluating papers, ultimately aiming to shift community norms towards more rigorous reporting practices. Read this article to gain insight into the ongoing efforts to enhance the integrity and reliability of scientific research.

Recent past and present The leading empirical psychology journal, Psychological Science, will begin requiring authors to disclose flexibility in data collection and analysis starting on January of 2014 (see editorial). The leading business school journal, Management Science, implemented a similar policy a few months ago.Both policies closely mirror the recommendations we made in our 21 Word Solution piece, where we contrasted the level of disclosure in science vs. food (see reprint of Figure 3). Our proposed 21 word disclosure statement was:We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.Etienne Lebel tested an elegant and simple implementation in his PsychDisclosure project. Its success contributed to Psych Science's decision to implement disclosure requirements.Starting Now When reviewing for journals other than Psych Science and Management Science, what could reviewers do?On the one hand, as reviewers we simply cannot do our jobs if we do not know fully what happened in the study we are tasked with evaluating.On the other hand, requiring disclosure from an individual article one is reviewing risks authors taking such requests personally (reviewers are doubting them) and risks revealing our identity as reviewers.A solution is a uniform disclosure request that large numbers of reviewers request for every paper they review.Together with Etienne Lebel, Don Moore, and Brian Nosek we created a standardized request that we and many others have already begun using in all of our reviews. We hope you will start using it too. With many reviewers including it in their referee reports, the community norms will change:I request that the authors add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes. The authors should, of course, add any additional text to ensure the statement is accurate. This is the standard reviewer disclosure request endorsed by the Center for Open Science [see http://osf.io/project/hadz3]. I include it in every review.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe

[10] Reviewers are asking for it

Data Colada

[98] Evidence of Fraud in an Influential Field Experiment About Dishonesty

A friend recently asked for my take on the Miller and Sanjurjo's (2018; pdf) debunking of the hot hand fallacy. In that paper, the authors provide a brilliant and surprising observation missed by hundreds of people who had thought about the issue before, including the classic Gilovich, Vallone, & Tverksy (1985 .htm).In this post: (1) I try to provide an intuitive explanation for the Miller & Sanjurjo critique, and (2) explain why I still believe in the hot hand fallacy [1].Definitions: hot hand, and hot hand fallacy Hot hand: players are more likely to make a shot after making some shots. Say, after three :                      hot hand:                  P(H|HHH)> P(H)                               notation: P(H|HHH) is read as "probability of making the next shot:  P(H, after 3 shots are made  |HHH)".Hot hand fallacy: people believe the hot hand is bigger than it is :                        hot hand fallacy:      P̂(H|HHH) > P(H|HHH)                                 notation: P̂( H|HHH) is read as "predicted probability of making the next shot:  P̂(H, after 3 shots are made  |HHH)".Importantly, both can coexist. For example, Steph Curry might convert 45% of his three-point shots in general, increasing to 50% after making two in a row, but people believe it increases to 60%.Claims about sports vs about thinking Gilovich et al 1985 made two central claims: (1) the hot hand is basically zero, and (2) people over-perceive hot hands. Miller and Sanjurjo found evidence consistent with the hot hand in Gilovich et al's data, contradicting claim (1); it is not wrong to believe in the hot hand to some degree. I am hopeful (as are they) that their findings will refocus the field's attention to claim (2) which I find more important for behavioral science.  This quote from Matthew Rabin (2002; .pdf), describing the phenomenon with the more accurate label "hot hand bias", captures the perspective I hope we will all move towards (p.805): Gilovich, Vallone, and Tversky [1985] […] demonstrated that, while basketball fans believe that basketball players are streak shooters […] such a hot hand does not in fact exist (or at least not nearly to the degree that people believe in it).  To be clear, Miller & Sanjurjo's discovery must be taken into account also when studying the second claim. I do so here. Hot hand artifact To determine if there is a hot hand, we must compare the players’ likelihood to make a shot after a streak, P(H|streak), to some benchmark. The benchmark that researchers have used is the overall probability of making a shot: P(H). This assumes that, in the absence of a “hot hand”, we should see P(H|Streak)=P(H).  Miller and Sanjurjo discovered that this is incorrect given the way researchers have defined “streak”. This leads researchers to under-detect true hot hands, and therefore to overestimate the magnitude of the hot-hand fallacy. I refer to this issue as the 'hot hand artifact' [2].To understand the hot hand artifact, we need to differentiate between streaks and substreaks:Streaks are sets of consecutive hits, starting with the first hit in the set. Substreaks are subsets of streaks, starting with any hit in the streak.A streak of 2 means there have been 2 hits in a row. A substreak of 2 means there have been at least 2 hits in a row.If you flip a coin 3 times and get Heads three times:There is only one streak of HH:         [1][2]There are two substreaks of HH:       [1][2] & [2][3]I next simply state the hot hand artifact, I explain it afterwards.Stating the hot hand artifact The hot hand artifact arises when researchers compute how likely a hit is after substreaks rather than streaks of hits.Let's do coins. If you look for streaks, heads is always 50%: P(H|HHH)=P(H|HH)=P(H|H)=P(H)=.5But if you look at substreaks, that's not true. After a substreak of HH, the probability of another H is <50%. That’s extremely nonintuitive, so let's go over it in baby-steps.Let's represent a substreak by putting a '__' before the Hs; thus a substreak of 3 is _HHH. Then, we have this definition of the hot hand artifact, with say 3 hits:            hot hand artifact: P(H|_HHH) < P(H)So to recap:             hot hand:             P(H|HHH) > P(H)             hot hand fallacy:  P̂(H|HHH) > P(H|HHH)             hot hand artifact: P(H|_HHH) < P(H) Initial intuition for why substreaks matter: Why P(H|_H)<P(H)? Imagine a weird coin that always comes up Heads 10 times in a row, then 1 Tails, then 10 Heads, then 1 Tails, etc.HHHHHHHHHHTHHHHHHHHHHTHHHHHHHHHHT…How likely is H, after 9 H in a row (i.e., P(H|HHHHHHHHH)? Depends on whether we mean a streak of (exactly) 9 or a substreak of (at least) 9.If we mean a streak of 9, the answer is "100%"; this weird coin always gets 10 H in a row. If we mean a substreak of 9, the answer is… …50%.When the substreak starts with the 1st H, it is 100% When the substreak starts with the 2nd H, it is 0% On average, 50%.Second baby step: a normal coin I now reproduce the 1st example in Miller & Sanjurjo 2018, doing the hot hand artifact calculations for a 50:50 coin tossed three times. I explain it a bit differently, but it is the same example (see their Table 1 .png). Tossing 3 coins there are 23=8 possible outcomes. Streaks:        After flipping exactly 1 Heads, 50% of the time we see Heads again. Substreaks:  After flipping at least 1 Heads, 42% of the time we see Heads again.The only difficult thing here is that 2nd row: HHT. If you understand that row, you understand the hot hand artifact. So let's focus on that row.In the middle table: starting with the first H, we can only get HH In the right table: starting with the first H, we also get HH, but starting with the 2nd H, we get HT: so P(H|_H)=.5. By expanding our search to substreaks, this second row lowers the share of HH from 100% to 50%. When we average this now lower row with the others, we obtain P(H|_H)<P(H). Here is another way of thinking about it. There are two stages. First, data are generated. There is no bias there, coin is unbiased. Second, we search the data for substreaks, and our search is biased. If we find an HT we stop the search, if we find HH we keep looking, thus giving HT more chances.How the hot hand artifact challenges the evidence of the hot hand fallacy Having understood streaks vs substreaks, and why P(H|_H)<P(H) let's see why this matters for hot hand research.Recall that: hot hand fallacy: P̂(H|HHH) > P(H|HHH)To study the hot hand fallacy, then, we need to compare P̂() and P(),perceived and real hand-hotness.We estimate the former from participants, asking: "when a player makes 3 in a row, how likely are they to make a 4th?". We estimate the latter from 'real' data (e.g., NBA data), and it is in the real data that the hot hand artifact matters. This is because researchers have looked for substreaks rather than streaks.Researchers looked at how often players hit a shot after 3 in a row, without restricting the search to having made only 3 in a row. Looking at substreaks of shots has led researchers to under-estimate real hand-hotness. This is simply a brilliant insight by Miller and Sanjurjo.Thus, the criticism is that hot hand fallacy researchers found this: P̂(H|HHH)          >              P(H|_HHH)But it could be that: P̂(H|HHH) = P(H|HHH) > P(H|_HHH)In words, it is possible that rather than people over-estimating hot hands, researchers under-estimated them.  For instance, Gilovich et al in one study estimate the true hot hand to be +3%, p = .49 (comparing shooting % after making vs missing 3 shots).  Miller & Sanjurjo correct for the hot hand artifact and put the estimate at 13% (with a wide CI: approx 4%-22%) [3].Why I still believe in the hot hand fallacy I still believe in the hot hand fallacy for many reasons: First, it follows naturally from lots of things we have learned since 1985, see e.g., the review by Alter & Oppenheimer (2006; .pdf). Second, I look around. People everywhere interpret 'trends' after seeing just two datapoints (in sports, politics, faculty meetings, etc.). Third, and most relevant, I still believe in the hot hand fallacy, because it is present in the classic 1985 hot hand paper even taking into account the hot hand artifact. I will discuss two findings.Finding 1. Two free-throws Gilovich et al. report that participants expected an NBA player with a 70% free-throw average to make 74% of shots after making one, and 66% after missing one. An 8% difference in the estimate of P̂(H|H) – P̂(H|M). Gilovich et al contrasted that 8% in perceived streakiness, with their estimated actual streakiness of NBA free-throws: 0% [4]  Because free-throws were analyzed in sets of two, there are no substreaks, no hot hand artifact. Miller & Sanjurjo discussed these data in their footnote 22, which I have transcribed, and commented on, in this footnote [5] Finding  2. 100 single-shot predictions. Another result by Gilovich et al which is not threatened by the hot hand artifact is Study 4, where 26 varsity players took 100 shots each, predicting each time if they would make the next shot (putting money on the line). While here we do look at substreaks, a previous hit may have been preceded by other hits, we are contrasting shots after substreaks with predictions after the same substreak, so the hot hand artifact cancels out.Players were much more likely to bet that they would make a shot after making rather than missing  the previous shot.  But they were wrong. Previous shots did not predict future shots. Overall, the correlation between making the previous shot and predicting making the next shot was r=.4, while the correlation between making the previous shot and actually making the next was just r=.05.   Predicted impact of previous shot:  r=.40 –  Actual impact of previous shot        r=.05 = hot hand fallacy.This finding was replicated in a sample of Israeli Olympiads, obtaining r=.45 & r=.06 respectively  (Avugos et al., 2013 .pdf) [6]My favorite fact for this post: calibration of massiveness of hot hand fallacy It is worth noting that expecting r=.40 for consecutive shots (see chart above) implies implausibly strong hand-hotness. As a calibration, I simulated a player that when cold (previous shot was missed) makes as few shots as the NBA player with the lowest shooting percentage in the 2019-2020 season (Taurean Prince: 37.6%), but then gets dramatically better with each basket that he makes until, after three consecutive hits, he acquires the best shooting percentage in the league, 74.2%, that of Mitchell Robinson. In a series of 100 shots, this impossibly hot-handed player would on average have a correlation of r=.17 for two consecutive shots (R Code). In the Gilovich et al study, and in the Israeli replication, people expected their hand-hotness to be more than twice this implausible upper bound. [7].I am using what participants predicted ("hit" vs "miss") as a proxy for the probability they assigned a hit or a miss. This is more of a leap of faith than seems at first, but I think a reasonable one. See footnote for details [8]  Conclusions The hot hand artifact discovered by Miller and Sanjurjo is extraordinarily nonintuitive and it was pursued in an extremely clever, thorough, and elegant manner. But, when we focus on the evidence supporting the hot hand fallacy, the psychologically interesting, and practically consequential finding that people misinterpret meaningless random noise as meaningful streaks that they fallaciously expect to persist, that evidence is not challenged by the hot hand artifact. Author feedback. Our policy (.htm) is to share drafts of blog posts that discuss someone else's work with them to solicit suggestions for things we should change prior to posting. I shared a draft with Tom Gilovich, who made some suggestions and expressed agreement with the emphasis on the psychologically interesting question of over-perception of streakiness. I also shared a draft with Adam Sanjurjo and Joshua Miller with whom I discussed several aspects of the analyses and additional papers that were relevant. The conversation was polite, constructive, interesting, and very pleasant. In response to this conversation I added footnotes 1 & 8.  After our conversation two key disagreements remained. In their words:  "While we believe that Uri's characterization of Gilovich et al 1985's two claims that '(1) there is no hot hand, and (2) the hot hand is overperceived' is technically correct, it perhaps fails to convey exactly what the original, and many subsequent, authors long took 'overperceive' to mean. In particular, to them it meant perceiving something that does not exist, not, for example, perceiving something more than one ought to, or more frequently than one ought to. Second, we believe that the bets placed in Gilovich et al (1985)'s study are only minimally informative about what bettors' subjective probability estimates might be."Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe Footnotes.In conversations with Adam Sanjuarjo he persuaded me the term hot hand bias, which they used in earlier versions of the paper, would be more accurate than fallacy, but he also persuaded people would confuse 'hot hand bias' with the selection bias they discovered; so I ended up sticking with fallacy. [↩]Miller & Sanjurjo refer to it as the streak selection bias, but given the definition of 'streak' I use in this post, such name would be confusing. [↩]I obtained an approximate confidence interval for the 13% estimate by resampling under the null, obtaining a width of ~18%, which I apply symmetrically around the estimate of 13%, so 13+-9.   R Code [↩]That 0% estimate came from just 9 players; but since then, it has been estimated to be in the 0-2% range using data from the entire league in a few different papers, 2% is much less than the perceived 8%. See next footnote for details. [↩]Footnote 22 by Miller and Sanjurjo on Finding 1: illusory hot hand in free-throws. I paste here their footnote 22 followed by my take on it. By GVT they mean the original hot hand fallacy paper by Gilovich, Vallone & Tversky 1985 I tend to agree with (i) and (ii), but my interpretation is different. The fact that we should not rationally expect one free throw to predict the next, means that free-throws are indeed not an ideal test of the sports question of whether there is a hot hand in basketball, but for that same reason they are a good test of the psychology question of whether there is a hot hand fallacy.  Even in a setting where streakiness should not be expected, people expect it. It's a conservative test, and the hot hand fallacy passes it with some comfort. In terms of (iii), I would add some nuance to the characterization that "subsequent studies … find evidence that is inconsistent with the conclusions that GVT give" (bold added). Specifically, the subsequent studies do challenge the claim that there is no hot hand at all, but support the claim that people over-perceive it:Goldman & Rao (2012): I did not find a result in the paper that spoke to P(H|H) vs P(H|M).Aharoni and Sarig (2011) write "As in GVT, we find no simple dependency between the outcomes of first and second FTs." (p.2311).Arkes 2010 (this Arkes is not Hal Arkes), finds about <3% hot hand.Miller & Sanjurjo (2014), same authors, previous paper, write "we find that the performance after a hit is not significantly better than after a miss in six out of seven seasons, with a difference of one percentage point or less in most seasons (though the difference is significant when all seasons are pooled together). P10.Yaari Eisenman (2011) finds an effect in the 2% rangeWardrop (1995) "the hot hand phenomenon is not supported by these free throw data" (page 27)So my take of the follow-up work is that, with much larger samples than those used by Gilovich et al., the hot hand effect in free throws is in the 0-2% range, but people believe it is at about 4 times larger. Because 8%>2%, there is a hot hand fallacy.  [↩]Avugos et al report a second replication with recreational players, finding r=.249 and r=.002 for one variation, and r=.289 & r=.044 for the other. [↩]Miller and Sanjurjo have a working paper looking at this study in GVT. In a nutshell they argue, correctly in my view, that the analyses are underpowered to detect true hand-hotness.  The calibration provided above is indeed consistent with that. But, I would add, the analyses are even more underpowered to detect people's perception of their own hand hotness. If people had calibrated beliefs about sequential dependency of their shots we should not be able to detect any sequential dependency in their predictions; but we do, big time. This goes back to the research question. These data are useful to study the psychological question of the hot hand fallacy, and not useful to study the sports question of the hot hand. [↩]On making inferences about beliefs from observed  predictions . It is somewhat tricky to infer what participants believe from what they predict. Imagine someone who always predicts 'hit' if the expected probability is p>.5, and always predicts 'miss' otherwise. If the baseline is at P̂(H)=50%, then a tiny change in beliefs following a hit, say, going from 49.9% to 50.1%, could have a big effect in what participants predict. Adam Sanjurjo and Josh Miller brought this valid concern up in our conversations. There are three reasons I am not super concerned about over-interpreting this result (correlation of .40  vs .05 in Study 4). First, in practice, people don't make predictions in that bang-bang way. People don't predict 100% of hits, when they think the probability is 50.1%; rather, people 'probability match' (e.g., see  review by Vulkan 2000; htm). That is, when people think something has an x% chance of happening, they predict it x% of the time. Therefore, in practice, the more often people predict something the higher they perceive its probability to be. Second, if people did make bang-bang  predictions, attenuation, rather than exaggeration of measured hot hand fallacy seems likely. Unless all participants perceive the probability to be extremely close to 50% (or whatever cutoff people used for always saying "hit"or "miss"), then even large changes in beliefs do not change what people predict. e.g., if upon seeing someone score, the perceived probability of them scoring again goes from 35% to 45%, the prediction of "miss" is not changed. Third, in the Cornell data, there is some variation across players in their hit rate, ranging from making 25% to 61% of their shots across the 26 players. Yet we see a very strong correlations for previous shot made and next prediction for all of them, even those who are far from 50%. Moreover, there is no apparent association between distance from the 50% average and strength of the correlation between past shot and prediction for next shot:  Thus while the inference rests on the assumption that people predict 'hit' more often if they think a 'hit' is more likely, the assumption seems reasonable. [↩]

[88] The Hot-Hand Artifact for Dummies & Behavioral Scientists

In this sixth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JCR) article entitled, “The Impact of Resource Scarcity on Price-Quality Judgments” (.html).This one was full of surprises.The primary thesis of this article is straightforward: “Scarcity decreases consumers’ tendency to use price to judge product quality.” It reports six studies, three of which were conducted on MTurk. We chose to replicate Study 2A (N = 149) because it was the MTurk study with the strongest evidence, and because it also tested the effects of scarcity on two other measures: desire for abundance and categorization tendency (described below).In this study, participants were randomly assigned to complete a writing task that was designed to either induce a feeling of scarcity or not. They then rated their desire for abundance, completed a task designed to measure their categorization tendency, and then judged the quality of 10 computer monitors based solely on their prices. The estimated correlation between price and quality served as the authors’ primary dependent variable. Writing about scarce resources increased the desire for abundance (p = .006), lowered categorization tendency (p < .001), and, most importantly, reduced the judged correlation between quality judgments and prices (p = .020) [1].We contacted the authors to request the materials needed to conduct a replication. Unfortunately, the first author could not locate the original Qualtrics file, but he did send us a document containing a detailed summary of the methods. In general, the authors were forthcoming and polite, doing their best to answer all of our questions about how to best design our replication. We are very grateful to them for their help and professionalism.The ReplicationIn our replication, we tried our best to follow the procedures the authors outlined for us. There were, however, necessarily a few deviations, which we detail in this footnote: [2]. Also, in collaboration with the original authors, we tried our best to use exclusion criteria that made sense. You can access our Qualtrics surveys here (.qsf), our materials here (.pdf), our data here (.csv), our codebook here (.xlsx), and our R code here (.R).In this study, participants began by completing a writing task borrowed from Roux, Goldsmith, and Bonezzi (2015; .pdf). This served as the critical manipulation of scarcity. You’re going to want to look at this manipulation closely, because it affected participants in surprising ways:After completing this task, participants completed three dependent measures.Measure #1: Desire For AbundanceFirst, we measured their Desire For Abundance, which consisted of three items rated on a 7-point scale (1 = Not at all; 7 = Definitely). Those items were: “I desire to have a lot of things”, “I desire to own a lot of things”, and “Having a lot of things makes me happy.” These ratings were averaged.Measure #2: Categorization TendencyParticipants then completed a measure that the authors dubbed “Categorization Tendency.” To assess this, participants were first shown these four shapes:They were then asked to rate how much they agreed or disagreed with the following statements (1 = Strongly Disagree and 7 = Strongly Agree):I would not group object a) with either c) or d).I would not group object d) with either a) or b).Object d) is dissimilar to both a) and b).Object a) is dissimilar to both c) and d).Responses to these four statements were reverse-coded and then averaged, so that higher scores indicated a greater tendency to disagree with them.Before moving on, and to foreshadow some of what comes next, it is worth thinking about what this measure is capturing. On the one hand, it might represent a meaningful measure of a person’s tendency to categorize things. On the other hand, it might operate as an attention check, as it is arguably strange for a person who comprehends these statements to strongly agree with them. For example, it is arguably strange to strongly agree that a black square is dissimilar to both a black circle and a white square. Indeed, the modal response to all four statements was “Strongly Disagree,” while the least popular response was “Strongly Agree.”Measure #3: Price-Quality CorrelationAfter completing the categorization task, participants were asked to rate the quality of 10 computer monitors given only their prices. Each monitor was displayed on its own screen. Here’s what it looked like:There were 10 monitors, all priced differently. The prices, which were presented in one of four possible random orders, were $75, $99, $165, $190, $260, $295, $325, $330, $345, and $460. The key measure in this study was the within-subject correlation between participants’ quality ratings and the monitors’ prices [3].Replication ResultsAs in the original study, those who did the Scarcity writing task exhibited a lower price-quality correlation than those who did the Control writing task. This effect was much smaller than in the original study – dreplication = .15 vs. doriginal = .42 – but it was nevertheless significant: In addition, although the writing task did not significantly influence Desire For Abundance (d = .08), it did have a very significant effect on Categorization Tendency (d = .22):So although our observed effects were much smaller than in the original study, this looks like at least a partial success. . .Except, there is a very big problem:The above graph shows all of the dropouts and pre-registered exclusions in each of the two conditions. You can see that participants who encountered the Scarcity writing task were much more likely to drop out than those who encountered the Control writing task. This differential attrition means that random assignment was compromised, that the people who completed the measures in the Scarcity condition were probably different from those who completed the measures in the Control condition [4].OK, but how might they be different?Well, it turns out that good participants – those who follow instructions and honestly answer questions – were more likely to drop out of the Scarcity condition. Surprised? Yeah, us too. But here’s how to make sense of it.Imagine Lisa is a good, honest participant, while Mr. Burns is a bad, corruptible participant who just wants to get paid. If they are assigned to the Control condition, they are asked to write three things they did last week. This is simple enough for both Lisa to do (honestly) and for Mr. Burns to do (honestly or dishonestly), so both of them do it. But if they are assigned to the Scarcity condition, they are asked to write about three times in which resources were scarce. Maybe this is prohibitively difficult for them to do honestly but easy for them to do dishonestly, in which case Lisa bails but Mr. Burns stays. The result is that the Scarcity condition has (1) fewer people overall and (2) a disproportionate share of Mr. Burnses.Now, in the context of this study, how can having more bad participants in the Scarcity condition cause the results described above? It will be helpful to answer this question with data. To provide that data, Joe (painstakingly) coded all of the written responses submitted by participants. Participants who sensibly followed the instructions were coded as “Good”; participants who did not were coded as “Bad” [5].Here are the price-quality correlation results, broken down by Good vs. Bad Responders:Whoa.Bad Responders, who comprise 23.4% of the Scarcity condition sample but only 11.1% of the Control condition sample, exhibit much much smaller price-quality correlations than do Good Responders. Why? Recall that in the experimental task, participants were asked to judge the quality of computer monitors based only on their prices. There’s really only one way to do this. Good Responders should say that higher priced monitors should be of higher quality, thereby exhibiting a strong price-quality correlation. Many Bad Responders, on the other hand, may provide senseless quality judgments, causing their price-quality correlations to be much lower.Now, when you combine the fact that Bad Responders make up a disproportionate share of the Scarcity condition with the fact that Bad Responders are, by virtue of their sometimes meaningless responses, more likely to exhibit small price-quality correlations, you achieve the overall result, whereby those in the Scarcity condition exhibit smaller price-quality correlations. Here’s a picture that illustrates this:OK, but what about the other measures? Same story.You can see that Bad Responders are much more likely to score highly on Desire For Abundance and low on Categorization Tendency. And in the case of Categorization Tendency, the higher proportion of Bad Responders in the Scarcity condition is enough to produce a highly significant effect overall.Again, this makes sense. Recall that the Categorization Tendency measure is reverse-coded, and that it is comprised of four items that most humans will not agree with. Thus, most people who fully comprehend these items should score low on Categorization Tendency [6]. The Good Responders don’t agree with them, but the Bad Responders do, probably because they default to answering at the high end of rating scales (a possibility that would also explain the Desire For Abundance results).So, what have we learned?As has been recently emphasized (.html), differential attrition is a real thing to worry about, especially in studies in which one task is even slightly more onerous, difficult, or weird than another.Although differential attrition means that we cannot validly compare Good Responders in the Control condition to Good Responders in the Scarcity condition, it is worth noting that there is no hint of the authors’ hypothesized effects when we look only at Good Responders. In general, true effects should get stronger when selecting for more compliant, higher quality responders. In this study, they got weaker.Nevertheless, the jury is still out because a true test of this hypothesis would require conducting a study that eliminates the problem of differential attrition.We actually tried to do this [7]. In a pre-registered study (https://aspredicted.org/md5q8.pdf; N = 2,101), we randomly assigned half of our participants to a condition in which they were (1) warned they would have to do an onerous vowel-counting task as well as the writing task, (2) explicitly asked whether they wanted to continue despite that, and (3) actually did the vowel-counting task. This all happened before they were assigned to the Control vs. Scarcity writing task. Our hope was that Bad Responders would be caught before the writing task, either because they dropped out of the study or because they performed poorly on the vowel counting task (in which case we preregistered to exclude them). A full writeup of this study is here: .pdf.Alas. On the one hand, we were successful at dramatically reducing attrition. On the other hand, we were unsuccessful at reducing differential attrition, as once again many more participants dropped out of the Scarcity condition than the Control condition.So then we gave up, and decided to call the whole thing “inconclusive.”It turns out that eliminating differential attrition is hard. And why shouldn’t it be? Research is hard. Life is hard.Author feedbackWe appreciate the importance of the replication effort and are glad you are doing this work. We thank you for being so cooperative during the preparation process. We are also happy that most of our effects emerged as hypothesized in your replication study.A close look at your results has led us to identify a couple of important issues. First, what constitutes a “good” and “bad” responder? This is an immensely important (and difficult to answer) question that impacts a tremendous amount of behavioral research. The criteria you applied to the replication study showed a disproportionate number of bad responders in the scarcity (vs. control) condition, thus creating a confound between “participant goodness” and resource scarcity as potential drivers of our documented effects. To further address this issue, we went back to our original data and re-examined participant responses across conditions. First, we looked at criteria you used as indicators of bad responders to see if they varied as a function of scarcity vs. control condition. However, in our data, the percentage of participants giving poor quality responses in the writing task did not differ across conditions (p=.50). Also, the average time spent completing the writing task (p=.87) and the variance of the time spent on the writing task (p=.63) did not differ by condition. Another indicator of bad responders, the percentage of participants giving the same quality rating to all 10 computer monitors also did not differ across conditions (p=.29, although we should note that these numbers were very small). Further, if there are disproportionately more “bad” responders in the scarcity (vs. control) condition, it seems reasonable to expect greater variance in participant responses in the scarcity (vs. control) condition due to bad responders giving answers that are not contingent on experimental stimuli (i.e., more random and noisy). However, neither the variance of price-quality correlations (p>.99), desire for abundance (p=.52), nor categorization tendency (p=.12) differed in the scarcity (vs. control) condition.Participants’ motivations for quitting when confronted with the scarcity writing task is also worth further discussion. One possibility, as you suggest, is that the scarcity writing task “is prohibitively difficult for [participants] to do honestly but easy for them to do dishonestly,” and thus that “bad” (vs. “good”) responders are more likely to stay (vs. drop out) in the scarcity condition. It is also possible, however, that bad responders might also have more difficulty within the narrower response domain in the scarcity condition (e.g., generating a fake scarcity experience that they never had). Also, given that scarcity is a pervasive aspect of human life (Booth 1984; Lynn 1991), genuine scarcity-related experiences might be easily accessible to good responders. In any case, we appreciate you bringing up this valuable point.Reading this web article has given us an increased appreciation of the importance of the differential attrition phenomenon and the impact it can have on experimental findings. We applaud the Data Replicada team’s efforts to illustrate that phenomenon and we will definitely be more aware of this issue in the future. We encourage other researchers to carefully consider this as well."Footnotes.The authors also reported successful mediation, but we pre-registered not to conduct these mediational analyses. [↩]First, on one page of the original survey the authors showed participants a table that displayed 12 computer monitors, alongside a number of attributes, including brand name, country of origin, model year, screen size, price, and quality rating. The original study was conducted in 2017, so we felt we should update this table so as to show participants contemporary products and prices. Importantly, however, we did preserve the observed price-quality correlation depicted in the original table, as it was .6977 in the original study and .6980 in ours (this .pdf shows the two tables side-by-side). (A special thanks to our excellent research assistant Louise Lu, for helping us update these stimuli.) Second, whereas the original authors paid MTurkers no more than 20 cents to complete the survey, we were compelled to pay 50 cents, given how long it took to complete the survey. Third, the authors weren’t certain about the exact criteria they used to recruit participants, so we had to use our best judgment. Finally, during the quality judgment task, the original authors presented the 10 to-be-judged computer monitors (and their prices) in a fixed order, but they had no record of what that order was. We decided to create four different orders and to randomly assign participants to see one of those orders. [↩]In this study, we also randomly assigned one-fifth of our participants to complete a manipulation check that assessed their perceptions of resource scarcity instead of these three measures. Those participants (n = 120 in the Control condition and n = 76 in the Scarcity condition after pre-registered exclusions) are not included in the sample sizes reported in this post, and are necessarily excluded from all other analyses (since they did not complete any additional measures). We did find that the scarcity manipulation increased self-reported scarcity perceptions, t(194) = 2.22, p = .027. However this effect was quite a bit smaller than what the authors reported in their Web Appendix P: d = .33 in our replication vs. d = .69 in the original study. [↩]In the original study, there were 78 participants in the Control condition and 71 in the Scarcity condition. This discrepancy in per-condition sample sizes suggests that the original authors may have encountered differential attrition as well, though it may have been a smaller problem than what we’ve encountered here. [↩]A few notes about the coding. First, Joe was necessarily not blind to condition (or hypothesis) while doing the coding, but he was blind to participants responses’ to the dependent measures. Also, the vast majority of these judgments were completely unambiguous, like when respondents just wrote “good” when describing a scarce event in detail. Joe’s resulting “BadResponse” variable is in the posted dataset if you want to take a look. [↩]We changed these sentences on July 22, 2020, one day after publication. They used to read "…items that, really, no human being should agree with. Thus, no one who fully comprehends these items should score low on Categorization Tendency." We received an email from a human being who did agree with two of the items, and so our characterization was clearly mistaken. We appreciate that feedback, and apologize for the earlier mischaracterization. [↩]We actually ran two more studies. The first of these additional studies only investigated responses to the categorization tendency dependent variable. We randomly assigned participants (N = 1,556) to complete this measure either before or after being assigned to the Control vs. Scarcity condition. We were wondering whether the pattern of differential attrition would be so pronounced as to cause condition differences on the categorization tendency variable even before participants were assigned to condition. Instead, although we observed differential attrition in this study, we observed no effect of condition on categorization tendency, regardless of whether that measure was completed before or after the condition assignment. In other words, we did not replicate the significant effect observed here. For a full writeup of this study, click here: .pdf. [↩]

[89] Data Replicada #6: The Problem of (Weird) Differential Attrition

A number of authors have recently proposed that (i) psychological research is highly unpredictable, with identical studies obtaining surprisingly different results, (ii) the presence of heterogeneity decreases the replicability of psychological findings. In this post we provide evidence that contradicts both propositions.Consider these quotes:"heterogeneity persists, and to a reasonable degree, even in […] Many Labs projects […] where rigid, vetted protocols with identical study materials are followed […]  heterogeneity […] cannot be avoided in psychological research—even if every effort is taken to eliminate it."         McShane, Tackett, Böckenholt, and Gelman (American Statistician 2019 .htm) “Heterogeneity […] makes it unlikely that the typical psychological study can be closely replicated”         Stanley, Carter, and Doucouliagos (Psychological Bulletin 2018 .htm)"Repeated investigations of the same phenomenon [get] effect sizes that vary more than one would expect […]  even in exact replication studies. […] In the presence of heterogeneity, […] even large N studies may find a result in the opposite direction from the original study. This makes us question the wisdom of placing a great deal of faith in a single replication study"         Judd and Kenny (Psychological Methods 2019 .htm)This post is not an evaluation of the totality of these three papers, but rather a specific evaluation of the claims in the quoted text. In the case of the paper by Judd and Kenny, the claims quoted above are secondary, used to motivate the remainder of their paper (e.g., in the opening lines of their abstract).Heterogeneity: observable vs not. To begin, it is important to distinguish between two types of heterogeneity:Observable heterogeneity: When studies that look different obtain surprisingly different results.Unobservable heterogeneity: When studies that look identical obtain surprisingly different results. Note: by "surprisingly" we mean "beyond what's expected by sampling error alone."In what follows, we do two things.First, we report new data demonstrating that observable heterogeneity across studies does not imply low replicability. Second, we revisit the data behind the quotes above, finding there is no support for the conclusions that unobservable heterogeneity is rampant, or even present, in studies with identical designs (i.e., the Many Labs studies).(You can reproduce all of the results in this post: links to materials, data, and code: .pdf.)New Data: Maluma and Takiti Walk Into a Bar (Chart). Our first demonstration is based on a classic Gestalt psychology study, first reported by Wolfgang Köhler in 1929. He presented participants with two shapes, one with jagged edges and one with rounded "edges," as in the gray figures above. He then asked them which one should be called “Maluma” and which one should be called “Takiti.” The vast majority of participants said that the rounded shape must be “Maluma.” This effect was gigantic in the 1920s, and it is still gigantic today. It replicates nicely in class [1] [2].We conducted 50 variations of this "Maluma-Takiti" paradigm, but instead of using the words “Maluma-Takiti,” we used a Dutch boy name paired with a Dutch girl name. Specifically, we got a list of the 50 most popular baby names in the Netherlands in 2016 (source: .htm) and we paired the first name of the girl’s list with the first name on the boy’s list, and then the second name on the girl’s list with the second name on the boy’s list, and then….you get the idea. For example, our first study pitted the most popular boy name, Daan, against the most popular girl name, Anna, as follows [3].We ran the first wave of 50 studies in June of 2017. In total, we asked 508 U.S. MTurk participants to complete 10 studies each during a single session, and so there were about 100 observations per study [4]. We expected girl names to be assigned to the rounded shape more often than the boy names, but that this effect would vary substantially across name pairs. The studies came out as expected. Figure 1. 50 MTurk studies run in June 2017 (N=508). Each participant rated a random subset of 10 name-pairs, indicating, on a 1-5 scale, the extent to which one name or the other seemed more likely to be the name of the rounded shape (R Code).For example, the right-most bar shows that Nora (vs. Thijs) was much more likely to be assigned to the round shape (d = 0.71), while the left-most bar shows a slight reversal for Liz (vs. Luca) (d = -0.05).Running a meta-analysis on these 50 studies, we find that the standard measure of heterogeneity, I2, was quite high, at I2 = 78%, p < .0001 [5]. As a benchmark, consider that Stanley et al., quoted in the intro, computed an average I2=74% across 200 meta-analysis in psychology and wrote that such high heterogeneity makes it "unlikely that any replication will be successful" (p. 1339).To see whether such extreme pessimism is warranted, we ran all 50 studies again in our Wave 2, about three weeks ago (April 2019), nearly 2 years after Wave 1. The results replicated about as well as can be rationally expected. That is, they varied only as much as would be expected because of sampling error (see Figure 2). For example, if there were zero unobservable heterogeneity, we would expect effect sizes to be correlated r = .79 across waves. Our observed correlation was rather close to that, r = .77 (see footnote for calculations: [6]). Figure 2. Scatterplot of effect size estimates for 50 name-pairs shown in Figure 1, for two separate MTurk waves (R Code).When we asked them for feedback on this post, some of the authors we quoted did not think the scatterplot above looks so great for replicability, as there is quite a bit of noise between two estimates of the same study. We are not disputing the reality of sampling error, and it is a fact that studies with just n=100 are noisy. The question is whether they are noisier than expected by chance alone. The answer is 'no.' Perhaps this is easier to appreciate in Figure 3:Figure 3.  For each of the 50 studies we computed the absolute difference in effect size between waves. The expected (blue) line depicts the average differences in bootstrapped samples when forcing homogeneity, ranked from smallest to largest expected difference.  There is zero evidence of unobserved heterogeneity in the MTurk data (R Code)  These results show that while effect sizes were observably quite heterogeneous, unobservable heterogeneity was between nil and trivial, so properly powered replications of these designs would generally be expected to replicate.These results also show that MTurk is a reliable source of data, providing similar results when the same studies are collected 2 years apart.Note from 2019 04 25: After receiving an email inquiry, we have re-run all analyses dropping participants who partook in both waves, results are unchanged see footnote for details [7].Are identical studies really heterogeneous? Though we found that our 50 MTurk studies conducted two years apart were as stable as one should rationally expect, McShane and colleagues reported high instability in the MTurk results of a Many Labs paper run just days apart.For example, they wrote: “treating each [MTurk] day as a separate study […] yields seven extremely close replications […] Again, however, […] heterogeneity is nontrivial: […] nonzero estimates of heterogeneity for nine of the 14 non-null phenomena and across these, [I2= 21%]"       McShane, Böckenholt, and Hansen (2016); .htmNow, we understand that an I2 of 21% might seem “nontrivial,” or even high, but it is important to keep in mind that even if there were zero unobservable heterogeneity in MTurk across days, we would expect I2 to be greater than zero. There are two reasons for this. First, I2 is, well, a squared value, and squared values can’t be negative. Thus, it is possible for I2 to be greater than zero just by chance, but it can never be less than zero just by chance. Thus, on average, in the presence of zero heterogeneity, we would expect I2 to be greater than zero [8]. Second, I2 is easily biased upwards when you violate its distributional assumptions, and we have shown that this bias is present for the specific data analyzed by McShane et al. (http://datacolada.org/63).To take both issues into account, we bootstrapped the I2 in the Many Labs data, in order to ascertain what the I2 should be if there were truly no heterogeneity across studies. It turns out we should expect an average I2 of 24%. Recall that we observe slightly less variation than that: I2=21%. The null of homogeneity is not rejected: p = .44 (R Code). The observed amount of heterogeneity is trivial after all.The Many Labs MTurk data, then, is consistent with our Maluma-Takiti demonstration, and provides no evidence for the presence of hidden moderators.Many Labs Beyond MTurk The papers we quoted at the beginning of this post cite a few other Many Labs papers to support the notion that unobserved heterogeneity is a big problem in psychology. For example, McShane et al. mention that the many labs papers by (i) Eerland et al. 2016 and (ii) Hagger et al. (2016) show “moderate amounts of heterogeneity” (p. 101). Kenny and Judd (2018) mention those same two Many Labs papers as evidence of heterogeneity as well. Let’s look at them.Eerland et al. (.htm) analyzed three dependent variables. The first shows a high I2 but the null of zero heterogeneity is not terribly inconsistent with the data (p=.045). The other two DVs in that paper show zero heterogeneity (I2 = 0%, with p=.393 and p=.662).Hagger et al. (.htm) used two dependent variables. The p-values for heterogeneity were p=.045 and p=.056.Thus, the five p-values supporting “moderate amounts of heterogeneity” are .045, .045, .056, .662, and .393. We think most impartial observers would agree that this is not strong evidence.Importantly, there are at least six other Many Labs papers whose heterogeneity results are not mentioned in the context of these arguments. None of them exhibit significant heterogeneity (links to all papers in table: .pdf).Note: While writing this post we found a coding error in the heterogeneity calculations in Many Labs 3 (Ebersole et al 2016). See footnote for details: [9].There is also the recently published Many Labs 2 (.htm). For this blogpost we attempted to re-analyze the posted data, but we obtained substantially different results. We are insufficiently confident on the results to comment on them further at this point. Conclusions The evidence does not support the general claim that unobservable heterogeneity is inevitable in psychology, nor the specific claim that identical designs across different labs, or across time on MTurk, obtain surprisingly heterogeneous results.One theme in the original papers we quoted from is that heterogeneity makes it challenging to generalize effect size estimates from one design to another design. We fully agree with this. Moreover, we agree with Judd and Kenny that it is valuable to engage in "stimulus sampling," to include variations of stimuli within individual experiments. (If you are unfamiliar with stimulus sampling, please check out this important paper: .htm).But generalizability and replicability are different concepts. To establish that a finding is a scientific fact, you need to establish that it replicates under some specifiable condition(s). But you do not need to establish that it generalizes. For example, if I claim to have shown that people can levitate when they drink vanilla lattes, then I need to show that they can levitate under some set of predictable and reproducible conditions (i.e., when they drink vanilla lattes), but I do not need to show that they can levitate under other conditions (e.g., when they drink cappuccinos). Of course, whether they can also levitate when they drink cappuccinos is an interesting and important question, one that has implications for our understanding of why and how levitation might occur. But it makes no sense to ask whether my levitation finding is generalizable if I cannot first establish that it is replicable. Replicability is always the first thing that needs to be established, because without it there is no finding to explain or understand (see Colada[53]). We believe that these authors have conflated replicability and generalizability, and that that conflation lies at the root of our disagreement with them.The question of interest in this post is whether, in the presence of observable heterogeneity across different experimental designs, single-study direct replications can be informative if they are properly powered. For example, can we ex ante predict that people will assign Nora to the rounded shape more often than Thijs? Can we ex ante predict that the Nora-Thijs difference will be greater than the Liz-Luca difference?Our read of the quoted papers is that they say "no." Our read of the evidence is that it says "yes." Author feedback Our policy (.htm) is to share drafts of blog posts that discuss someone else’s work with them to solicit suggestions for things we should change prior to posting. We contacted all authors of the original quoted in the introduction, and the first author of Many Labs 3. We heard back from all teams. In response to their feedback, we corrected a few errors (specifically to some of the I2 p-values we were copy-pasting), added a bootstrap analysis of Maluma that draws participants rather than single ratings at random, added Figure 3, and made a few other clarifications throughout. We also revised the writing in an attempt to clarify the overall message of each of their papers, and why we ran the analyses that we did. We thank all of them for responding to our request for feedback. After a few email exchanges with Chick Judd and Dave Kenny, we identified the following issue that's at the heart of our disagreement. As they put it in an email to us "For us (Judd & Kenny), inevitably any attempt at replication involves generalization because the exact same situation, samples, and so on cannot be recreated.  Some of these changes likely moderate the effect, and even possibly there might be fundamentally random variation in the effect. The result is heterogeneity, even in ‘exact’ replications. We agree that the evidence for heterogeneity is weak in the case of null or essentially zero effects. The evidence is considerably stronger for non-null effects.”  We (Joe and Uri) disagree with them because to us the evidence available so far suggests the consequences of such minimal levels of unobservable heterogeneity are basically nil, thus 'exact' replications are often, in practice, obtained (e.g., our Maluma demonstration). All 4 of us agreed to include this summary.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe Footnotes.When writing this post we learned, with some consternation, that Köhler, in his original study, spelled Takiti as “Takete”. For a German in the late 1920s, we code this as a minor transgression. Actually, writing this footnote led us to look up Wolfgang Köhler (.htm). He was a vocal opponent of the Nazi regime and emigrated to our beloved Pennsylvania in the 1930s. He was a Swarthmore professor until 1955. [↩]Although Köhler gets the credit for this work, we have to give a shout out to Yale’s Shane Frederick for popularizing it in the JDM community. He is the reason we know about this. [↩]We counterbalanced the order of boy vs. girl names across the 50 studies [↩]The Many Labs papers, currently used as the best estimates of unobservable heterogeneity in psychology, also had participants complete multiple studies. [↩]This means that 78% of the variance across studies is beyond what's expected by chance. I2 is a bad measure of heterogeneity (see Colada[63]), but we use it anyway because the authors whose work we are commenting on used it [↩]To create the expected correlation between studies in the absence of heterogeneity, we shuffled observations between Waves 1 and 2 within each study, 10,000 times. This forces the null of no heterogeneity on the data, while preserving most other relevant data structures. The average correlation of these reshuffled datasets was r = .789. The figure below contrasts the observed with the expected correlation under the null. Figure A1. Distribution of correlations between two waves of the same 50 studies under the null that there is zero unobserved heterogeneity within study design across time. The observed correlation, r=.77,  is close to the center of the distribution expected under the null. If the bootstrap is done by drawing only from Wave 1 to create Waves 1 and 2, the expected correlation is similar: r=.786. If the bootstrap is done by drawing participants at random, so all the ratings of a given participant bundled for any given draw, taking into account the lack of independence across observations by the same participant, the expected correlation is unchanged: r=.795. See R Code Section #3.8 [↩]There are 36 MTurk IDs that appear on both waves, these are probably repeat participants. If we drop them (from both waves) the results are as follows. I2 in wave 1 goes from 78.4% to 76.7%. Correlation of Wave 1 and Wave 2 effects goes from r=.773 to r=.790 [↩]It is possible to obtain I2=0, but the probability of obtaining I2>0 when they is zero heterogeneity is typically greater than 0.40. See R Code. [↩]There are two significant results among the 13 they report, but that's due to a coding error, where the signs of a few estimates were mistakenly reversed, thus artificially inflating the estimates of heterogeneity.  The authors are working on a correction. In our Table 1, we report the the results as published (p = .17). The correct p-value is substantially higher than p =. 17. [↩]

[76] Heterogeneity Is Replicable: Evidence From Maluma, MTurk, and Many Labs

The authors of a forthcoming AER article (.pdf), "Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics", painstakingly harvested thousands of test results from 25 economics journals to answer an interesting question: Are studies that use some research designs more trustworthy than others? In this post I will explain why I think their conclusion that experiments in economics are less p-hacked than are other research designs is unwarranted.Disclaimer: I reviewed this paper for AER. The authors conclude:"our results suggest that the [instrumental variables] and, to a lesser extent, [difference-in-difference] research bodies have substantially more p-hacking and/or selective publication than those based on [randomized controlled trials] and [regression-discontinuity]" (p.3)The authors base these conclusions on three main analyses: "We use three approaches to document the differences in p-hacking […]: [1] Test for discontinuities […] just above or below a conventional statistical threshold […] [2] [compare the […] proportion of tests that are marginally significant […] [3] comparing the observed distribution of test statistics […] to [the distribution] we expect to emerge absent p-hacking and publication bias." (p.2&3)In this post I will focus on approaches 1 & 2, which are basically the same for our purposes (approach 3 is discussed towards the end). Approaches 1 & 2 conclude there is more p-hacking when there are more results just below vs. above a significant cutoff (e.g., more p=.049s than p=.051s). The figure below is a modified version of their Figure 2, showing histograms of collected z-values for two of the four literatures they review (recall, p=.05 when Z=1.96). (For this post I added colored bars, panel titles, and the legend. See the original AER figure .png)As the authors note, it is true that if there were no selective reporting, the red and blue bars should be about equally high. They are also right that the fact that those bars aren't equally high implies that these literatures exhibit selective reporting.It may be intuitively compelling to take one more logical step and assume the following: the more unequal the bars are, the more p-hacked a literature is. The authors rely on that assumption to conclude that experiments in economics are less p-hacked. But, the assumption is not correct.Fast vs slow p-hacking. Leif Nelson, Joe Simmons, and I coined the term p–hacking in our first p-curve paper (2014 | .htm). In Supplement 3 of that paper (.pdf), we modeled p-hacking as researchers computing, instead of just one p-value, a sequence of p-values. So researchers observe the p-value from the first attempted analysis, the p-value from the second attempted analysis, and so on. We used that framework to construct the expected distribution of p-values when a researcher engages in p-hacking. For this post all you need to know is that some forms of p-hacking change p-values more quickly than others.Slow p-hacking arises when the p-value changes by a small amount from analysis to analysis. Say you run a regression with 25,000 observations and obtain p=.09. If you drop 57 observations and re-run the regression, the p-value will change, but probably by a small amount. You have basically the same data, run with the same statistical tool. When p-values change slowly, p-hacking will cause p-values to bunch below p=.05. The intuition is straightforward: if you approach p=.05 in baby steps, when you cross .05 you won't land very far from it (≤ 1 baby step away). In studies with observational data, most available forms of p-hacking are probably slow: excluding a few observations here, transforming a variable there, expanding the before period by 2 weeks over there, etc.  More generally, to analyze messy observational data we have to make lots of small operationalization decisions; most of these decisions will have small consequences on the final result, so p-hacking these decisions will usually constitute slow p-hacking. My intuition and non-systematic evidence suggest that slow p-hacking is more common in observational studies; but it is an unanswered empirical question.Fast p-hacking arises when the p-value changes substantially from analysis to analysis. Say you run an experiment seeking to nudge people into vacuuming their couch more often. This is your one chance to run the couch-nudge study, so you include 14 manipulations (weekly vs. monthly reminders, opt-in/opt-out, gain/loss frames, etc.). The p-value comparing each pair of conditions will change substantially from pair to pair because the alternative tests are statistically independent. If one pair of conditions (say high vs. low sunk cost) obtained p=.72, another pair (e.g., vivid vs. dull images of couch-vacuuming) could easily be p=.001. Indeed, because p-hacking condition-pairs involves random draws from the uniform distribution (under the null), the p-value for two independent tests moves by 33 percentage points (from say p=.36 to p=.03, or vice versa) on average. My intuition and non-systematic evidence suggest that fast p-hacking is more common in experimental studies, especially costly field experiments; but it is an unanswered empirical question.p-hacking fast and slow, a simulation. To illustrate the association between fast vs. slow p-hacking and bunching below .05, I ran simulations where researchers use either slow p-hacking (highly correlated sequences of p-values, r=.75), or fast p-hacking (r=0). In both scenarios, p-hacking is equally consequential, increasing the false-positive rate from 5% to about 30%. Thus, both literatures are equally trustworthy (R Code).For reasons explained earlier, p-hacking causes bunching only with slow p-hacking.  While both literatures have been p-hacked to the same extent (same consequences), the analyses from the AER paper would conclude the first literature is much more p-hacked and much less trustworthy than the second.The third approach So far I have focused on the first two approaches the AER paper used to assess p-hacking. The third method, recall, consisted of…[3] comparing the observed distribution of test statistics […] to [the distribution] we expect to emerge absent p-hacking and publication bias." (p.3) This comparison of expected and observed distributions leads to striking visualizations where publication bias is evident to the naked eye. For instance, the figure below is a slightly modified version of their results for the difference-in-differences literature. The 'missing' results below Z=2 really jump out.(modified version of the 1st panel in AER paper's Figure 4 – see original .png)Unfortunately your naked eyes are deceiving you.  This figure provides no evidence of publication bias because the 'expected' line is not really what we should expect.Explaining this requires getting into the weeds, so you have to opt-in to read it.I opt-in to get into the weedsWhat should we expect? We could perhaps reasonably expect most individual test statistics in a paper to be distributed t (thanks to the central limit theorem). But we are plotting the distribution of all results, a distribution combining other distributions, known as a 'mixture', and the mixture of many t-distributions is not itself t (Wikipedia page on mixtures of distributions .htm).This sounds abstract and nit picky, which is precisely why I would not bother discussing it if it did not matter a lot. It has quite severe consequences. Bear with me.Perhaps the most intuitive problem with the 'expected' used, is that it is unimodal (one hump) but when we combine multiple tests from multiple studies, we will often get a bimodal distribution (two humps), or possibly multimodal. If we run the analysis in the AER paper as if we expected one hump, when we should actually expect two humps, we will generate spurious evidence of publication bias.Combining two normals → NOT normal For a quick intuition let's say half the tests in the literature are placebo tests with a mean effect of zero, and the other half examine effects of policies that actually work, say with a mean test result of Z=4. Let's start easy, using normal distributions. Say published placebo results come from N(0,1) and published treatment tests from N(4,1). If we combine all published results, placebos and tests, into one chart, what do we expect? We do NOT expect a normal, we do not expect N(2,SD).We expect two-humps, like this:Calibration for differences-in-differences chart If you go back to the with observed vs expected lines, you will see the tails are longer than from a normal. So instead of two normal distribution: 1.For the tests with zero true effect lets use t-distribution with df=2. Red line below. 2.For the tests of real policy changes (median Z about 4) one with df=1. Blue line below [1]. The literature that includes both types of tests is the sum of (1) & (2) lines: the actually expected distribution in this example: black line below.That expected black line looks a lot like the observed line in the expected vs observed chart motivating this section [2].But again, there is no publication bias here, we get the double hump because some effects are zero, and some are not zero. No single study has been dropped or p-hacked.Let's now treat the chart above as the data and try to fit, as in the AER paper, a single t- distribution to those 'data', forcing a unimodal distribution on data which are not unimodal. The figure below shows what would happen. Same true data with the same black line And adds the "expected" curve a-la AER, purple line.The figure seems to suggest that there are test results 'missing' left of 1.96, but they are only missing because we have the wrong model, not because somebody is hiding them.What I am and am not saying I am not saying the simple arbitrary structure in my calibration is the correct one. I am not saying the authors should have fit 2 instead of 1 distribution. I am saying that the conclusion that tests are missing from comparing observed with expected distributions of test results depend on the arbitrary structure chosen. While it is unclear, and I believe it is unknowable, what is the right structure to model, some structures are wronger than others.  Fitting a unimodal distribution to a mixture where we expect some effects to be zero and others not to be zero is almost surely wrong, and quite likely produces spurious publication bias. If we then use this approach to compare across literatures, which suffer from unobservably different specification error (due to unobservably different mixtures of distributions), it's unclear any signal can be possibly extracted from the data.R CodeConclusion I have proposed here that we should not infer that literatures with more bunching just past .05 are less trustworthy, and that visually striking comparisons of 'expected' and observed test results can be quite misleading due to incorrect assumptions about the expected line.  Fortunately, there is a statistical tool designed exactly for the question this AER paper is interested on, where the evidential value of different literatures can be compared: p-curve. P-curve has high power to detect evidential value and lack of it, it takes into account both slow and fast p-hacking, and it tells you not 'whether' there is selective reporting, but how strong the evidence for the effects of interest is once we control for selective reporting. I would not use p-curve, or any tool, however, to analyze 1000s of test results. The gains of a larger sample size become outweighed by the losses of interpretability and representativeness of any result.You don't need to know R to use p-curve, we made an online app for it: http://p-curve.com Author feedback. Our policy (.htm) is to share drafts of blog posts that discuss someone else's work with them to solicit suggestions for things we should change prior to posting. About 2 weeks ago I shared a draft with the authors of the article who wrote a response which included some questions for me. You can read their revised full response here: .pdf. Below I show the authors key comment/question in bold, followed by my comments.1. Authors:  How could slow vs fast p-hacking explain the differences in Instrumental Variables (IV) vs Regression Discontinuity Designs (RDD)? Uri: I have read too few articles of each type to have a strong view, but my intuition is two fold. First, RDD hinges more strongly on ad-hoc operationalization decisions than does IV research, such as functional form assumptions and bin-size definitions. These decisions tend to have large consequences (fast p-hacking). But more importantly, and separate from the issues in this post, different research designs lead to reporting different sets of results, making comparisons of the distributions of all published test results across designs difficult to interpret. For example, say IV papers tend to report more robustness tests, while RDD papers tend to report more placebo tests. If both literatures had equal selective reporting, or equal evidential value, the set of all published results would have different distributions, different mixtures.  The typical RDD paper in the AER sample had twice as many test results as the typical IV article did.  To give a concrete example, an RDD article in the sample examined the impact of close elections on violence in Mexico and reported 8 (null) results for before the election. These are effectively placebo tests which should be, and are, near zero. (The RDD paper is doi:10.1257/aer.20121637, the 8 null findings from its Table 2 are in the AER dataset, see article id #46).2. Authors: Should we really expect randomized control trials (RCT) to have more fast p-hacking? Uri: The authors gave four arguments of why they don't think that's the case. (1) RCT are more collaborative (more 'witnesses'), (2) are pre-registered, (3) reported to funders, and (4) difficult to just run and file-drawer whatever does not work. I think these are good arguments, and they gave me pause. I should make clear that I don't know that RCTs have more fast, or slow, p-hacking than do other methods. I guess nobody does. There are reasons to expect more p-hacking in experiments, and there are reasons to expect less p-hacking. Precisely because we don't know this, I object to relying on a statistical method that assumes we know this, and that we know it is identical across designs. To interpret bunching, we need to know how much fast and slow p-hacking happens. In addition, lots of p-hacking in the RCT literature is not hidden. Often RCT authors run many many tests, and report many tests, and then focus the discussion on the few significant ones. While transparent, this does not reduce the false-positive rate for the literature. (Some recent RCT studies control for some multiple comparisons.)3.  Authors: If we impose an assumption of a single round of fieldwork it would require that all the candidate nudges be tried at the same time, and the less favorable results then discarded. That seems to us a rather counterfactual to how RCTs actually work. Uri: I got a bit into this in the previous point (at least some p-hacking in RCT is out in the open). In addition, I would say that: (1) manipulations are but one approach to fast p-hacking. Others include choosing subgroups (analyze just boys or just girls) and measures. I would also say (2) pre-registrations do not 'prevent' p-hacking but 'simply' make it easier to identify it. For this blue paragraph I just went to the SocialScienceRegistry (what some economists use for 'pre-registration'), and searched for the name of a famous RCT researcher. Their first pre-registration to appear included the following eight variables as "primary" measures:   "income, consumption, assets, nutrition, health, food security, financial behaviors, and labor supply." Each, of course, is operationalizable in multiple ways, and many of them will be poorly correlated (fast p-hacking). In medicine, pre-registration has been required for clinical trials for years, and precisely thanks to those pre-registrations we know so many clinical trials are p-hacked (.htm).The authors full response, again, is available here .pdfSubscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe Footnotes.I set ncp=2.78 for this second curve, inducing a median effect of about Z=4 [↩](To be clear, that the example generates a chart that resembles Figure 4 in the AER paper is not a coincidence. I chose the parameters to obtain that shape. In order to explain how one can obtain Figure 4 in the absence of publication bias, I need to produce Figure 4 without publication bias. We do not generally expect that particular shape in real literatures, there is no particular shape that can be generally expected. Part of the problem is precisely this, it is not possible to generate, based on statistical principles alone, the expected distribution of test results [↩]

[91] p-hacking fast and slow: Evaluating a forthcoming AER paper deeming some econ literatures less trustworthy

In this second installment of Data Replicada, we report two attempts to replicate a study in a recently published Journal of Consumer Research (JCR) article entitled, “Wine for the Table: Self-Construal, Group Size, and Choice for Self and Others” (.htm).Imagine that you are in a monthly book club and it is your job to select books for the upcoming meetings. Further imagine that your book club members are equally divided between those who prefer fiction and those who prefer nonfiction. You personally prefer nonfiction. You could choose selfishly and select mostly nonfiction books, or you could less selfishly opt for a balanced reading list of fiction and nonfiction. In this kind of situation, when will you behave most selfishly?In this JCR paper, the authors propose that the answer to this question depends on two variables: (1) whether people are considering their independent selves or interdependent selves, and (2) whether the size of the group is small or large. Paraphrasing from the abstract, the key finding can be summarized thusly:When people are considering their interdependent selves, they consistently make choices that balance self and others’ preferences, regardless of group size. They are relatively unselfish.When people are considering their independent selves, they make choices that balance self and others’ preferences when the group is small, but they make choices that more strongly reflect their own preferences when the group is large.In other words, independent consumers who choose on behalf of large groups tend to choose more selfishly, whereas everyone else tends to choose less selfishly.The JCR paper contains six studies, all of which were conducted on MTurk. We chose to try to replicate Study 3 because it seemed to have the best statistical evidence, as it was supported by the smallest p-value in the paper: t(239) = 2.20, p = .03 [1]. In this study (N = 250), participants were first primed to think of their independent or interdependent selves. They then imagined being in a book club of either four or ten people, and were asked to choose which seven books the club members would have to read, on a scale ranging from zero nonfiction books (seven fiction books) to seven nonfiction books (zero fiction books). Subsequently, the researchers assessed participants’ own preferences for nonfiction vs. fiction books.The result was a three-way interaction among priming condition (independent vs. interdependent), group size condition (small vs. large), and participants’ relative preference for nonfiction books (continuously measured). When participants were primed with independence and imagined being in a larger book club, then their choice of whether to have the group read more nonfiction books was highly correlated with their own preference for nonfiction books. That is, they chose selfishly. But this correlation was smaller among participants who were primed with independence and imagined being in a small book club, and among participants who were primed with interdependence (regardless of the size of the book club).We contacted the authors to request the exact materials and the output of some of their analyses. They were extremely forthcoming and polite, quickly and thoroughly responding to all of our many follow-up questions and requests. We are extremely grateful to the authors for their help, professionalism, and patience.An Embarrassing InterludeWhile writing up the results of our replication, we realized that our survey contained a potentially important typo in a choice option that was part of the key dependent variable (described in more detail below). After discovering this mistake, we notified the original authors, and then, after much deliberation, decided to run a second replication. We report the results of both replications below.The ReplicationsIn our first attempt, we set out to run a preregistered replication that had about three times the sample size of the original (N = 800 MTurkers). We preregistered to adhere to the original authors’ exclusion criteria, but we severely underestimated the number of participants who would meet these criteria [2]. We wound up with only 565 participants and an inconclusive result. So we decided to get an additional 800 participants and to combine the data from these two waves for analysis [3]. We therefore have two preregistrations for this replication attempt (wave 1: https://aspredicted.org/ni9xq.pdf ; wave 2: https://aspredicted.org/zi4ue.pdf). After all exclusions we wound up with a sample of 1,118 participants, about 4.5 times the sample size of the original study.In our second attempt, we corrected the typo from the first attempt, and invited 2,000 unique MTurkers to participate. After exclusions, we wound up with 1,305 participants (https://aspredicted.org/gc74h.pdf).In our replications, we used the same instructions, procedures, questions, and participant population as in the original study [4] [5]. You can access our Qualtrics surveys here (.qsf1, .qsf2), our materials here (.pdf), our data here (.csv), our codebook here (.xlsx), and our R code here (.R).Participants completed a survey that was ostensibly comprised of two separate studies. The first “study” primed the independent vs. interdependent self by having participants count the number of pronouns in a paragraph that contained only singular pronouns (independent prime condition) or only plural pronouns (interdependent prime condition):In the second “study,” participants read a scenario in which they imagined belonging to a book club. The book club consisted of either three other members (small group condition) or nine other members (large group condition):Participants then learned that they were tasked with choosing books for the book club, that half of the book club prefers to read fiction and half prefers to read non-fiction, and to choose a book deal on behalf of the group. This was the key dependent variable, and this is where we bungled our first replication attempt:Instead of Bookseller 4 offering four fiction books and three non-fiction books, we accidentally presented participants with an offering of four fiction books and four fiction books. This typo meant that one option now presented participants with more books than all of the other options. We do not know how much this affected the results, but it is possible it affected them quite a bit. That is why we ran Replication 2 [6].After making this choice, participants then answered questions designed to assess how accountable they felt to others while making their choice, and then they indicated how often they read nonfiction books and how often they read fiction books. These three measures were used as covariates. Finally, participants indicated how much they like fiction vs. non-fiction books (on 7-point scales). We subtracted participants’ liking of fiction books from their liking of non-fiction books to compute a “preference for non-fiction” measure. In addition to the priming and group size manipulations, this represented the third key predictor in the main analysis.So, in sum, this study had three independent variables:Priming condition: independent vs. interdependent (manipulated)Group size condition: small vs. large (manipulated)Preference for non-fiction over fiction books (measured)The dependent variable is how many non-fiction books the participants chose on behalf of the book club (ranging from 0 to 7).ResultsHere are the original results, as well as our replication results. Keep in mind that in the graphs below, a more positive slope indicates greater selfishness.Let us first consider whether the original result was significant in the more highly powered replication attempts. In this case, the results are mixed. In the first replication attempt (with the typo), the three-way interaction among prime, group size, and preference for nonfiction was significant and in the same direction as in the original study: b = .162, SE = .070, p = .022. But in the second replication attempt (with no typo), that same interaction did not approach significance, though it was still in the same direction: b = .050, SE = .064, p = .432 [7] [8].Although the 3-way interaction is in the same direction as the original, the nature of that interaction is not perfectly consistent with the original authors’ theorizing. Recall that their theory holds that the independent/large-group condition will be more selfish than the other three conditions, and that it will therefore exhibit the strongest positive relationship between book preference and book choice. But in the first replication attempt, we find that participants in the independent/large-group condition are no more selfish than those in the interdependent/small-group condition. And in the second replication attempt, we find that those in the two interdependent prime conditions are more selfish than those in the independent/large-group condition. Thus, to the extent that we are finding evidence for the authors’ three-way interaction, its form seems to be somewhat different from what was hypothesized.Focusing again on the 3-way interaction, we can now consider what the replication results imply about the possible size of this effect, and about the sample size that would be required to detect it.If the true effect size is equal to what we observed in Replication 1, then the original study (N = 250) would have had 22.4% power to detect it, and you would need N = 1,650 to have an 80% chance to detect it. If the true effect size is equal to what we observed in Replication 2, then the original study would have had 7.8% power to detect it, and you would need N = 16,500 to have an 80% chance to detect it.In sum, our replications suggest that (1) the author’s predicted three-way interaction may or may not be a real effect, but (2) the effect is much smaller than what was reported in the original study, (3) the effect cannot be reliably observed using the original author’s sample size, and (4) the precise shape of this interaction is inconsistent across studies and not perfectly consistent with the authors’ theorizing. At least within this specific paradigm, there is not yet compelling evidence that those who are making decisions on behalf of a group are most selfish when the group is large and when they are considering their independent selves.Author feedbackWhen we reached out to the authors for comments on the post, they responded as follows:"Looking over the graphs, we were encouraged to see the same basic pattern appear – at least directionally – across the original study and both replications. In the first replication, the pattern appears to be strong despite any noise that resulted from the programming error. In the second replication, the pattern is directional but weaker. We speculate that the results of this second replication may have been dampened because the study was conducted shortly before Christmas (potentially affecting the self-construal prime and/or participants' choices), or perhaps because of the necessity of delving deeply into the mTurk pool in order to acquire the desired sample sizes."Across all three studies, however, our perspective is that there is indeed an interaction between self-construal and group size that drives consumers' choices for self and others, though the size of this effect is not clear. We thank the Data Replicada team for their interest in our work, and we hope that these results will encourage future replications and research in this space, so that we may better understand the factors that drive such choices." Footnotes.The other critical t-values were 1.87, 2.17, 1.99, 1.87, and 1.87, respectively. [↩]For a full description of how many participants we excluded and why, see this .pdf. [↩]Yes, this is a (fairly trivial) form of p-hacking; if we correct for it, then the critical p-value for Replication 1 is .047 instead of .022. [↩]There was only one noteworthy procedural difference. The original study contained a screen on which participants were presented with examples (and images) of newly released fiction and nonfiction books. Since that study was conducted in 2015 and ours was conducted in 2019, we had to decide whether to use the authors’ original books/images or to update them for 2019. We weren’t sure what to do, and so we decided to do both: half of the participants saw the original books/images and half saw books/images that we selected based on criteria the authors used to select their own stimuli. We collapse across this factor in the analyses presented below, but we should note that in our first replication the key result is only significant in the condition in which we presented the 2019 books. (In our second replication, the effect was somewhat stronger in the condition in which we presented the 2019 books: p = .396 vs. p = .780.) Perhaps it was important to update the books, or perhaps this is just chance. [↩]In the original study, the authors also conducted a test for mediation, using participants’ open-ended responses to one of the questions included in the survey. Although our data allow for us to conduct that mediation test, we have not done so, and in fact preregistered to ignore it. Interested readers can download the data and conduct that analysis themselves. [↩]We do know that this had at least some effect on the results: Participants were more likely to choose Bookseller 4’s offering in Replication 1 (73.1%) than in Replication 2 (60.5%), χ2(1) = 42.33, p < 10-10. [↩]The key analysis in this replication – a regression involving a three-way interaction among prime condition, group size, and preference for non-fiction books – has an inflated false-positive rate: 8.8% in Replication 1 and 7.2% in Replication 2. This is probably because of violations of assumptions that can emerge whenever a predictor is non-binary (e.g., outliers and/or heteroskedasticity), as in the case of the preference for nonfiction variable. We can fix it well enough if we use robust standard errors (Using robust standard errors reduced the false-positive rates to 5.2% in Replication 1 and 5.4% in Replication 2.). We do so in all analyses reported here, though we only preregistered this in Replication 2 (since we did not know about this problem before running Replication 1). Using robust standard errors appropriately increases the p-values that we report, but in this instance it did not change the significance of any of them. [↩]At the end of the study, we asked participants, “To your knowledge, have you ever participated in a past study on MTurk that asked you to imagine being in a bookclub and then to make choices on behalf of that bookclub?” (Possible answers: No, Maybe, Yes). Restricting the analyses to those participants who answered “No” to this question (N = 1,009 in Replication 1; N = 1,158 in Replication 2) makes the Replication 1 results a bit stronger (p = .004) and the Replication 2 results a bit weaker (p = .877). [↩]

[83] Data Replicada #2: Do Self-Construal and Group Size Influence How People Make Choices on Behalf of a Group?

In the tenth installment of Data Replicada, we report our attempt to replicate a recently published Journal of Consumer Research (JMR) article entitled, “Goal Conflict Encourages Work and Discourages Leisure” (.htm).The article’s two key hypotheses are right there in the title: People who are faced with a goal conflict are (1) more likely to spend time on work and (2) less likely to spend time on leisure. This is because “goal conflict increases reliance on salient justifications” and “work tends to be easier to justify and leisure harder to justify” (Abstract).This JCR paper contains seven studies, and three of those used MTurk participants. We chose to replicate Study 1a (N = 210) because it represented a simple test of both of the authors’ hypotheses. In this study, participants first listed two goals that they were currently pursuing (e.g., “lose weight” and “save money”; or, “grow a bigger garden” and “get a dog”). They were then randomly assigned to write about ways in which those goals either do or do not conflict. Following that task, participants were introduced to an ostensibly unrelated book-browsing task that was framed as either work or leisure, and the researchers recorded how much time the participants spent on this task. They found that participants in the goal conflict condition spent more time on the book-browsing task when it was framed as work and less time on the book-browsing task when it was framed as leisure.We contacted the authors to request the materials needed to conduct a replication. They were forthcoming, thorough, and polite. They immediately shared the original Qualtrics file that they used to conduct that study, and we used it to conduct our replication. They also promptly answered a few follow-up questions, and provided us with data that allowed us to better compare the results of our replications and their original study. We are very grateful to the authors for their help and professionalism.The ReplicationsWe ran two identical replications. The first was run on MTurk using the same criteria the original authors specified. The second was run on MTurk using a new(ish) feature that screens for only high-quality “CloudResearch Approved Participants” [1].In the pre-registered replications [2], we used the same survey as in the original study, and therefore the same instructions, procedures, stimuli, and questions. These studies did not deviate from the original study in any discernible way, except that our consent form was necessarily different, and our exclusion rules were slightly different [3]. After exclusions, we wound up with 1,008 participants in Replication 1 and 1,127 participants in Replication 2 (~5 times the original sample size). You can (easily!) access all of our pre-registrations, surveys, materials, data, and code on ResearchBox (.htm).In each study, participants were first asked to “list two of your goals.” Then, on the next page, they were randomly assigned to either “describe a time when you DID NOT FEEL conflict between these goals” (the No-Conflict condition) or to “describe a time when you felt conflict between these goals” (the Conflict condition).Participants then moved on to a separate book-browsing task, which was either described as work or leisure, as shown here:Participants then engaged in the book-browsing task itself. Here is how it looked:If participants clicked the box next to any of the books, the book’s description would appear below this array. They could spend as little or as much time on the task before proceeding, up to an undisclosed limit of 300 seconds, at which point the survey auto-advanced to the next screen.The primary dependent measure was the time participants spent on this screen.After this task, participants completed some manipulation check measures, an attention check, and some demographic questions.ResultsBefore presenting the results, it is worth discussing our analysis plan. The original authors offered and tested two independent hypotheses, and so we pre-registered to conduct separate tests of those hypotheses. Specifically, we examined whether participants who did vs. did not write about goal conflict (1) spent more time on the book-browsing task when it was framed as work, and (2) spent less time on the book-browsing task when it was framed as leisure.As in the original article, our critical dependent measure was the natural log of seconds spent on the book browsing task [4]. Here are the original vs. replication results:Let’s consider each of the two independent hypotheses.First, did writing about goal conflict increase the time spent on the book-browsing task when it was framed as work? Although the original authors reported fairly strong support for this hypothesis (d = .58), we found no evidence for it, despite having about five times more observations than in the original study. If this effect is real, it is very small, as we observed a small negative effect in Replication 1 (d = -.098) and a similarly sized positive effect in Replication 2 (d = .096). With only ~53 per cell, we estimate that the original authors had only 0.7% power to detect the effect size we observed in Replication 1 and only 7% power to detect the effect size we observed in Replication 2 [5]. You’d need ~1,700 per cell to have 80% power to detect an effect of the size that we observed in Replication 2 (i.e., d = .096).Second, did writing about goal conflict decrease the time spent on the book-browsing task when it was framed as leisure? Although this effect is not significant in either of our replications, we do find directional and marginally (or close-to-marginally) significant support for it in both. Nevertheless, the effect sizes we observed (ds = -.13 and -.15) are much smaller than what was reported in the original (d = -.34), again indicating that the original study had only 10-12% power to detect such an effect. You’d need ~800 per cell to have 80% power to detect effects of that size (i.e., ds of about -.15).On balance, then, these replications were not successful. If the effects exist, they could not have been detected in the original study [6].What About Data Quality?Although no one can ever definitively rule out that a replication failed because of either lower data quality or the mere passage of time, it is worth noting that our descriptive statistics and data quality seem comparable to the original study.First, we found that participants in our first replication – which sampled from the same population as the original study – averaged almost the same amount of time on the book-browsing task (M = 71.2 sec, SD = 69.7) as did participants in the original study (M = 73.2 sec, SD = 67.9) [7]. This suggests that our MTurk participants were approaching the task in a way that was similar to how MTurkers in the original study approached it [8].Second, after the book browsing task, participants answered questions designed to serve as a check on the effectiveness of the goal conflict manipulation. Just as in the original study, we found a highly significant effect of goal conflict on this measure (ps < 10-10).ConclusionIn sum, we did not find significant or consistent evidence that writing about a goal conflict increases time spent on a book browsing task framed as work. We did find marginally significant evidence that writing about a goal conflict decreases time spent on a book browsing task framed as leisure, but the effect, if it exists, is so small that the original study could not have reliably detected it.Author FeedbackWhen we reached out to the authors for comment, they had this to say:"We were encouraged to see the overall pattern of results from the original study emerge in the second, higher-quality replication sample, as well as in the leisure frame condition of the first sample (although with smaller effect sizes).Further, as footnoted in Data Colada’s post, in the second replication, the interaction predicted in the paper is also significant. Although we appreciate Data Colada’s rationale for interpreting our time spent hypothesis as two simple effects, not all experiments in the paper test such separate time spent on work versus leisure (i.e., consider relative time allocated to work versus leisure instead). We acknowledge that the wording of our paper title and time spent prediction resulted in some (unintended) ambiguity regarding germane tests of our theory.We also note that the context in which the replications were run—December 2020, at the height of the COVID-19 pandemic in the United States—may be relevant. Indeed, reviewing the data, the pandemic appears regularly in participant responses, with some even referencing it to counterargue the goal conflict manipulation. While we were glad the goal conflict manipulation check worked, the ongoing upheaval in participants’ daily lives and personal goals may have nevertheless impacted how they were influenced by the prompt. Moreover, the pandemic has created widespread change in how people both engage in and experience work and leisure (e.g., Giurge, Whillans, and Yemiscigil 2021; Kniffin et al. 2021; Lashua, Johnson, and Parry 2021; Sugar 2020, .html). Although the replication (consistent with the original study) does not include a work-leisure manipulation check, we surmise that sensitivity to the work-leisure manipulation (framing time spent on the same online book browsing task as a “leisurely” vs. “productive” activity) may be influenced by this broader context.Overall, we appreciate the Data Colada team’s interest in this research and support the replication effort. We hope this may increase interest in advancing understanding of goal conflict and its effect on time use."Footnotes.Specifically, in the first replication, we used MTurkers with at least a 96% approval rating. In the second replication, we used only “CloudResearch Approved Participants” with at least a 96% approval rating. As in the original, the survey software prevented mobile phone users from taking the survey. [↩]As always, we intended to pre-register our replications. However – and this is embarrassing – as we were preparing to put the files on Research Box after writing a draft of this post, we realized that we somehow forgot to submit our pre-registration answers, which we typed up in a Word document, on AsPredicted.org for the first replication. So we didn’t actually officially pre-register this study. Importantly, we did not know this until we completed all of our analyses, and, except for a minor deviation (reported below) that has zero consequences, we perfectly followed what we had decided to pre-register. We have uploaded the Word document containing this pre-registration to ResearchBox. If you look at its properties, you’ll see that we have not modified this document since December 1, 2020, two days before running Replication 1. So for all intents and purposes, this study was effectively pre-registered, despite our blunder. So we will refer to it as such. [↩]In keeping with the original authors, we excluded participants who failed an attention check or who spent more than 30 minutes on the survey. In addition, for quality control, we pre-registered to exclude all observations associated with duplicate MTurk IDs or IP Addresses, and to exclude those whose actual MTurk IDs were different from their reported IDs. [↩]As mentioned above, the time spent on the book browsing task was capped at 300 seconds by the book-browsing software. Nevertheless, a few participants spent a couple seconds more time than that, and one participant in our first replication somehow managed to spend more than 400 seconds on the task. (This participant also reported having some technical difficulties with the survey.) As a result, we winsorized all times exceeding 300 seconds down to 300 seconds prior to conducting our analyses. We did not pre-register this in our first replication, as we did not foresee the issue, but we did so in our second. The effects of this analytic decision are trivial. For example, if we do not winsorize in this way in Replication 1, the work-frame p-value becomes .264 instead of .270, and the leisure-frame p-value remains at .088. [↩]These power calculations reflect the probability of observing a significant effect that is the *same direction* as in the original study, using a two-sided test. This represents a departure from how we computed power in past Data Replicadas, but we think it is superior, both because we obviously care about the likelihood of finding an effect that is directionally consistent with the original study, and because this approach allows us to compute implied power even when the effect observed in the replication is in the opposite direction of the original. By this approach, the likelihood of finding a significant effect in the original direction if true d = 0 would be .05/2 = .025. The likelihood of finding a significant effect in the original direction if the true effect is in the *opposite* direction would be less than .025. [↩]As noted, we pre-registered to separately analyze the effect of goal conflict on time spent on the work-framed task and on time spent on the leisure-framed task. This is appropriate whenever the key prediction is a crossover interaction, because the interaction itself may not fully support that hypothesis. For example, an interaction can be significant even if only one or neither of the hypothesized simple effects has statistically significant support. Nevertheless, readers may be curious about the goal conflict x work/leisure-frame interaction in these studies. This interaction was significant in Replication 2, b = 17.60, SE = 8.55, t(1,123) = 2.06, p = .040, and non-significant in Replication 1, b = 8.35, SE = 8.61, t(1,004) = 0.97, p = .333. [↩]The medians were also very similar: 52.4 in the original study vs. 53.6 in Replication 1. [↩]The CloudResearch Approved participants in Replication 2 spent longer (M = 87.2 seconds, SD = 72.5, Median = 66.7), which is consistent with what we have been finding in the last few Data Replicadas: These participants take greater care when completing surveys. As further evidence, more participants failed the attention check (and were thus excluded) in Replication 1 (13.2%) than in Replication 2 (3.6%). [↩]

[97] Data Replicada #10: Does Goal Conflict Affect Time Spent on Work and Leisure?

A few years ago, we developed p-curve (see p-curve.com), a statistical tool that identifies whether or not a set of statistically significant findings contains evidential value, or whether those results are solely attributable to the selective reporting of studies or analyses. It also estimates the true average power of a set of significant findings [1].A few methods researchers have published papers stating that p-curve is biased when it is used to analyze studies with different effect sizes (i.e., studies with "heterogeneous effects"). Since effect sizes in the real world are not identical across studies, this would mean that p-curve is not very useful.In this post, we demonstrate that p-curve performs quite well in the presence of effect size heterogeneity, and we explain why the methods researchers have stated otherwise.Basic setup Most of this post consists of figures like this one, which report the results of 1,000 simulated p-curve analyses (R Code).Each analysis contains 20 studies, and each study has its own effect size, its own sample size, and because these are drawn independently, its own statistical power. In other words, the 20 studies contain heterogeneity [2].For example, to create this first figure, each analysis contained 20 studies. Each study had a sample size drawn at random from the orange histogram, a true effect size drawn at random from the blue histogram, and thus a level of statistical power drawn at random from the third histogram.The studies’ statistical power ranged from 10% to 70%, and their average power was 41%. P-curve guessed that their average power was 40%. Not bad.But what if…?1) But what if there is more heterogeneity in effect size? Let’s increase heterogeneity so that the analyzed set of studies contains effect sizes ranging from d = 0 (null) to d = 1 (very large), probably pretty close to the entire range of plausible effect sizes in psychology [3].The true average power is 42%. P-curve estimates 43%. Again, not bad.2) But what if samples are larger? Perhaps p-curve’s success is limited to analyses of studies that are relatively underpowered. So let’s increase sample size (and therefore power) and see what happens. In this simulation, we’ve increased the average sample size from 25 per cell to 50 per cell.The true power is 69%, and p-curve estimates 68%. This is starting to feel familiar.3) But what if the null is true for some studies? In real life, many p-curves will include a few truly null effects that are nevertheless significant (i.e., false-positives).  Let’s now analyze 25 studies, including 5 truly null effects (d=0) that were false-positively significant.The true power is 56%, and p-curve estimates 57%. This is continuing to feel familiar.4) But what if sample size and effect size are not symmetrically distributed? Maybe p-curve only works when sample and effect size are (unrealistically) symmetrically distributed. Let's try changing that. First we skew the sample size, then we skew the effect size:The true powers are 58% and 60%, and p-curve estimates 59% and 61%. This is persisting in feeling familiar.5) But what if all studies are highly powered? Let's go back to the first simulation and increase the average sample size to 100 per cell:The true power is 93%, and p-curve estimates 94%. It is clear that heterogeneity does not break or bias p-curve. On the contrary, p-curve does very well in the presence of heterogeneous effect sizes.So why have others proposed that p-curve is biased in the presence of heterogeneous effects? Reason 1:  Different definitions of p-curve’s goal. van Aert, Wicherts, & van Assen (2016, .htm) write that p-curve “overestimat[es] effect size under moderate-to-large heterogeneity” (abstract). McShane, Bockenholt, & Hansen (2016 .htm) write that p-curve “falsely assume[s] homogeneity […] produc[ing] upward[ly] biased estimates of the population average effect size.” (p.736).We believe that the readers of those papers would be very surprised by the results we depict in the figures above. How can we reconcile our results with what these authors are claiming?The answer is that the authors of those papers assessed how well p-curve estimated something different from what it estimates (and what we have repeatedly stated that it estimates).They assessed how well p-curve estimated the average effect sizes of all studies that could be conducted on the topic under investigation. But p-curve informs us “only” about the studies included in p-curve [4].Imagine that an effect is much stronger for American than for Ukrainian participants. For simplicity, let's say that all the Ukrainian studies are non-significant and thus excluded from p-curve, and that all the American studies are p<.05 and thus included in p-curve.P-curve would recover the true average effect of the American studies. Those arguing that p-curve is biased are saying that it should recover the average effect of both the Ukrainian and American studies, even though no Ukrainian study was included in the analysis [5].To be clear, these authors are not particularly idiosyncratic in their desire to estimate “the” overall effect.  Many meta-analysts write their papers as if that’s what they wanted to estimate. However…•  We don’t think that the overall effect exists in psychology (DataColada[33]). •  We don’t think that the overall effect is of interest to psychologists (DataColada[33]). •  And we know of no tool that can credibly estimate it.In any case, as a reader, here is your decision: If you want to use p-curve analysis to assess the evidential value or the average power of a set of statistically significant studies, then you can do so without having to worry about heterogeneity [6].If you instead want to assess something about a set of studies that are not analyzed by p-curve, including studies never observed or even conducted, do not run p-curve analysis. And good luck with that.Reason 2: Outliers vs heterogeneity Uli Schimmack, in a working paper (.pdf), reports that p-curve overestimates statistical power in the presence of heterogeneity. Just like us, and unlike the previously referenced authors, he is looking only at the studies included in p-curve. Why do we get different results?It will be useful to look at a concrete simulation he has proposed, one in which p-curve does indeed do poorly (R Code):  Although p-curve overestimates power in this scenario, the culprit is not heterogeneity, but rather the presence of outliers, namely several extremely highly powered studies. To see this let's look at similarly heterogeneous studies, but ones in which the maximum power is 80% instead of 100%.In a nutshell the overestimation with outliers occurs because power is a bounded variable but p-curve estimates it based on an unbounded latent variable (the noncentrality parameter). It's worth keeping in mind that a single outlier does not greatly bias p-curve. For example, if 20 studies are powered on average to 50%, adding one study powered to 95% increases true average power to 52%, p-curve's estimate to just 54%.This problem that Uli has identified is worth taking into account and perhaps p-curve can be modified to prevent such bias [7]. But it is worth keeping in mind that this situation should be rare, as few literatures contain both (1) a substantial number of studies powered over 90% and (2) a substantial number of under-powered studies. Moreover, this is somewhat inconsequential mistake. All it means is that p-curve will exaggerate how strong a truly (and obviously) strong literature actually is.In Summary •  P-curve is not biased by heterogeneity. •  It is biased upwards in the presence of both (1) low powered studies, and (2) a large share of extremely highly powered studies. •  P-curve tells us about the study designs it includes, not the study designs it excludes.Author feedback. Our policy (.htm) is to share, prior to publication, drafts of posts with original authors whose work we discuss, asking them to identify anything that is unfair, inaccurate, misleading, snarky, or poorly worded.We contacted all 7 authors of the three methods papers discussed above. Uli Schimmack declined to comment. Karsten Hansen and Blake McShane provided suggestions that led us to more precisely describe their analyses and to describe more detailed analyses in Footnote 5. Though our exchange with Karsten and Blake started and ended with, in their words, "fundamental disagreements about the nature of evaluating the statistical properties of an estimator,” the dialogue was friendly and constructive. We are very grateful to them, both for the feedback and for the tone of the discussion. (Interestingly, we disagree with them about the nature of our disagreement: we don't think we disagree about how to examine the statistical properties of an estimator, but rather, about how to effectively communicate methods issues to a general audience).  Marcel van Assen, Robbie van Aert, and Jelte Wicherts disagreed with our belief that readers of their paper would be surprised by how well p-curve recovers average power in the presence of heterogeneity (as they think their paper explains this as well). Specifically, like us, they think p-curve performs well when making inferences about studies included in p-curve, but, unlike us, they think that readers of their paper would realize this. They are not persuaded by our arguments that the population effect size does not exist and is not of interest to psychologists, and they are troubled by the fact that p-curve does not recover this effect. They also proposed that an important share of studies may indeed have power near 100% (citing this paper: .htm). We are very grateful to them for their feedback and collegiality as well.Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe Footnotes.P-curve can also be used to estimate average effect size rather than power (and, as Blake McShane and Karsten Hansen pointed out to us, when used in this fashion p-curve is virtually equivalent to the maximum likelihood procedure proposed by Hedges in 1984 (.htm) ).  Here we focus on power rather than effect size because we don't think "average effect size" is meaningful or of interest when aggregating across psychology experiments with different designs (see Colada[33]). Moreover, whereas power calculations only require that one knows the results of the test statistic of interest (e.g., F(1,230)=5.23), effect size calculations require one to also know how the study was defined, a fact that renders effect size estimations much more prone to human error (see page 676 of our article on p-curve and effect size estimation (.htm) ). In any case, the point that we make in this post applies at least as strongly to an analysis of effect size as it does to an analysis of power: p-curve correctly recovers the true average effect size of the studies that it analyses, even when those studies contain different (i.e., heterogeneous) effect sizes. See Figure 2c in our article on p-curve and effect size estimation (.htm) and Supplement 2 of that same paper (.pdf)  [↩]In real life, researchers are probably more likely to collect larger samples when studying smaller effects (see Colada[58]). This would necessarily reduce heterogeneity in power across studies [↩]To do this, we changed the blue histogram from d~N(.5,.05) to d~N(.5,.15). [↩]We have always been transparent about this. For instance, when we described how to use p-curve for effect size estimation (.htm) we wrote, "Here is an intuitive way to think of p-curve’s estimate: It is the average effect size one expects to get if one were to rerun all studies included in p-curve." (p.667).  [↩]For a more quantitative example check out Supplement 2 (.pdf) of our p-curve and effect size paper. In the middle panel of Figure S2, we consider a scenario in which a research attempts to run an equal number of studies (with n = 20 per cell) testing either an effect size of d = .2 or an effect size of d = .6. Because it necessarily easier to get significance when the effect size is larger than when the effect size is smaller, the share of significant d = .6 studies will necessarily be greater than the share of significant d = .2 studies, and thus p-curve will include more d = .6 studies that d = .2 studies. Because the d = .6 studies will be over-represented among all significant studies, the true average efefct of the significant studies will be d = .53 rather than d=.4. P-curve correctly recovers this value (.53), but it is biased upwards if we expect it to guess d=.4. For an even more quantitative example, imagine the true average effect is d = .5 with a standard deviation of .2. If we study this with many n=20 studies, the average observed significant effect will be d=.91, but the true average effect of those studies is d = .61, which is the number that p-curve would recover. It would not recover the true mean of the population (d = .5) but rather the true mean of the studies that were statistically significant (d = .61). In simulations, the true mean is known and this might look like a bias. In real life, the true mean is, well, meaningless, as it depends on arbitrary definitions of what constitutes the true population of all possible studies; R Code) [↩]Again, for both practical and conceptual reasons, we would not advise you to estimate the average effect size, regardless of whether you use p-curve or any other tool. But this has nothing to do with the supposed inability of p-curve to handle heterogeneity. See footnote 1. [↩]Uli has proposed using z-curve, a tool he developed, instead of p-curve.  While z-curve does not seem to be biased in scenarios with many studies with extreme high-power, it performs worse than p-curve in almost all other scenarios. For example, in the examples depicted graphically in this post, z-curve’s expected estimates are about 4 times further from the truth than are p-curve's. [↩]

[67] P-curve Handles Heterogeneity Just Fine

Are hidden moderators a thing? Do experiments intended to be identical lead to inexplicably different results?Back in 2014, the "Many Labs" project (htm) reported an ambitious attempt to answer these questions. More than 30 different labs ran the same set of studies and the paper presented the results side-by-side. They did not find any evidence that hidden moderators explain failures to replicate, but did conclude that hidden moderators play a large role in studies that do replicate.Statistically savvy observers now cite the Many Labs paper as evidence that hidden moderators, "unobserved heterogeneity", are a big deal. For example, McShane & Böckenholt (.htm) cite only the ManyLabs paper to justify this paragraph "While accounting for heterogeneity has long been regarded as important in meta-analyses of sets of studies that consist of […] [conceptual] replications, there is mounting evidence […] this is also the case [with] sets of [studies] that use identical or similar materials.” (p.1050) Similarly, van Aert, Wicherts , and van Assen (.htm) conclude heterogeneity is something to worry about in meta-analysis by pointing out that "in 50% of the replicated psychological studies in the Many Labs Replication Project, heterogeneity was present” (p.718)In this post I re-analyze the Many Labs data and conclude the authors substantially over-estimated the importance of hidden moderators in their data.Aside:  This post was delayed a few weeks because I couldn't reproduce some results in the Many Labs paper.  See footnote for details [1].How do you measure hidden moderators? In meta-analysis one typically tests for the presence of hidden moderators, “unobserved heterogeneity”, by comparing how much the dependent variable jumps around across studies, to how much it jumps around within studies (This is analogous to ANOVA if that helps).  Intuitively, when the differences are bigger across studies than within, we conclude that there is a hidden moderator across studies.This was the approach taken by Many Labs [2]. Specifically, they reported a statistic called I2 for each study. I2 measures the percentage of the variation across studies that is surprising.  For example, if a meta-analysis has I2=40%, then 60% of the observed differences across studies is attributed to chance, and 40% is attributed to hidden moderators.Aside: in my opinion the I2 is kind of pointless. We want to know if heterogeneity is substantial in absolute, not relative terms. No matter how inconsequential a hidden moderator is, as you increase sample size of the underlying studies, you will decrease the variation due to chance, and thus increase I2. Any moderator, no matter how small, can approach I2=100% with a big enough sample. To me, saying that a particular design is rich in heterogeneity because it has I2=60% is a lot like saying that a particular person is rich because she has something that is 60% gold, without specifying if the thing is her earring or her life-sized statue of Donald Trump. But I don't know very much about gold. (One could report, instead, the estimated standard deviation of effect size across studies ‘tau’ / τ). Most meta-analyses rely on I2. This is in no way a criticism of the Many Labs paper.Criticisms of I2 along these lines have been made before, see e.g., Rucker et al (.htm) or Borenstein et al (.pdf).I2 in Many Labs Below is a portion of Table 3 in their paper: For example, the first row shows that for the first anchoring experiment, 59.8% of the variation across labs is being attributed to chance, and the remaining 40.2% to some hidden moderator.Going down the table, we see that just over half the studies have a significant I2. Notably, four of these involve questions where participants are given an anchor and then asked to generate an open-ended numerical estimate.For some time I have wondered if the strange distributions of responses that one tends to get with anchoring questions (bumpy, skewed, some potentially huge outliers; see histograms .png), or the data-cleaning that such responses led the Many Labs authors to take, may have biased upwards the apparent role of hidden moderators for these variables.For this post I looked into it, and it seems like the answer is: ay, squared.Shuffling data. In essence, I wanted to answer the question: “If there were no hidden moderators what-so-ever in the anchoring questions in Many Labs, how likely would it be that the authors would find (false-positive) evidence for them anyway?”To answer this question I could run simulations with made up data. But because the data were posted, there is a much better solution: run simulations with the real data.   Importantly, I run the simulations “under the null,” where the data are minimally modified to ensure there is in fact no hidden moderator, and then we assess if I2 manages to realize that (it does not). This is essentially a randomization/reallocation/permutation test [3].The posted “raw” datafile (.csv | 38Mb) has more than 6000 rows, one for each participant, across all labs. The columns have the variables for each of the 13 studies. There is also a column indicating which lab the participant is from. To run my simulations I shuffle that "lab" column. That is, I randomly sort that column, and only that column, keeping everything else in the spreadsheet intact. This creates a placebo lab column which cannot be correlated with any of the effects. With the shuffled column the effects are all, by construction, homogenous, because each observation is equally likely to originate in any “lab.” This means that variation within lab must be entirely comparable to variation across labs, and thus that the true I2 is equal to zero.  When testing for heterogeneity in this shuffled dataset we are asking: are observations randomly labeled “Brian’s Lab” systematically different from those randomly labeled “Fred’s Lab”? Of course not.This approach sounds kinda cool and kinda out-there. Indeed it is super cool, but it is as old as it gets. It is closely related to permutation tests, which was developed in the 1930s when hypothesis testing was just getting started. [4].Results. I shuffled the dataset, conducted a meta-analysis on each of the four anchoring questions, computed I2, and repeated this 1000 times (R Code).  The first thing we can ask is how many of those meta-analyses led to a significant, false-positive, I2.  How often do they wrongly conclude there is a hidden moderator?The answer should be, for each of the 4 anchoring questions: "5%”. But the answers are: 20%, 49%, 47% and 46%.Yikes.The figures below show the distributions of I2 across the simulations.Figure 1: For the 4 anchoring questions, the hidden-moderators test Many Labs used is invalid: high false-positive rate, inflated estimates of heterogeneity (R Code). For example, for Anchoring 4, we see that the median estimate is that I2=38.4% of the variance is caused by hidden moderators, and 46% of the time it concludes there is statistically significant heterogeneity. Recall, the right answer is that there is zero heterogeneity, and only 5% of the time we should conclude otherwise. [5],[6],[7].Validating the shuffle. To reassure readers the approach I take above is valid, I submitted normally distributed data to the shuffle test. I added three columns to the Many Labs spreadsheets with made up data: In the first, the true anchoring effect was d=0 in all labs. In the second it was d=.5 in all labs. And in the third it was on average d=.5, but it varied across labs with sd(d)=.2 [8].Recall that because the shuffle-test is done under the null, 5% of the results should be significant (false-positive), and I should get a ton of I2=0% estimates, no matter which of the three true effects are simulated.That’s exactly what happens.Figure 2: for normally distributed data, ~5% false-positive rate, and lots of accurate I2=0% estimates (R Code). I suspect I2 can tolerate fairly non-normal data, but anchoring (or perhaps the intensive way it was 'cleaned') was too much for it. I have not looked into which specific aspect of the data, or possibly the data cleaning, disconcerts I2 [9].Conclusions. The authors of Many Labs saw the glass half full, concluding hidden moderators were present only in studies with big effect sizes. The statistically savvy authors of the opening paragraphs saw it mostly-empty, warning readers of hidden moderators everywhere. I see the glass mostly-full: the evidence that hidden moderators influenced large-effect anchoring studies appears to be spurious.We should stop arguing hidden moderators are a thing based on the Many Labs paper.Author feedback Our policy is to share drafts of blog posts that discuss someone else’s work with them to solicit feedback (see our updated policy .htm). A constructive dialogue with several authors from Many Labs helped make progress with the reproducibility issues (see footnote 1) and improve the post more generally. They suggested I share the draft with Robbie van Aert, Jelte Wicherts and Marcel Van Assen and I did. We had an interesting discussion on the properties of randomization tests, shared some R Code back and forth, and they alerted me to the existence of the Rucker et al paper cited above.In various email exchanges the term "hidden moderator" came up. I use the term literally: there are things we don't see (hidden) that influence the size of the effect (moderators), thus to me it is a synonym with unobserved (hidden) heterogeneity (moderators). Some authors were concerned that "hidden moderator" is a loaded term used to excuse failures to replicate. I revised the writing of the post taking that into account, hopefully making it absolutely clear that the Many Labs authors concluded that hidden moderators are not responsible for the studies that failed to replicate. Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe  Footnotes.The most important problem I had was that I could not reproduce the key results in Table 3 using the posted data. As the authors explained when I shared a draft of this post, a few weeks after first asking them about this, the study names are mismatched in that table, such that the results for one study are reported in the row for a different study.  A separate issue is that I tried to reproduce some extensive data cleaning performed on the anchoring questions but gave up when noticing large discrepancies between sample sizes described in the supplement and present in the data. The authors have uploaded a new supplement to the OSF where the sample sizes match the posted data files (though not the code itself that would allow reproducing the data cleaning). [↩]they also compared lab to online, and US vs non-US, but these are obviously observable moderators [↩]I originally called this "bootstrapping," many readers complained. Some were not aware modifying the data so that the null is true can also be called bootstrapping. Check out section 3 in this "Introduction to the bootstrap world" by Boos (2003) .pdf. But others were aware of it and nevertheless objected because I keep the data fixed without resampling so to them bootstrapping was misleading. Permutation is not perfect because permutation tests are associated with trying all permutations. Reallocation test is not perfect because they usually involve swapping the treatment column, not the lab column. This is a purely semantic issue, as we all know what the test I run consisted of. [↩]Though note that here I randomize the data to see if a statistical procedure is valid for the data, not to adjust statistical significance [↩]It is worth noting that the other four variables with statistically significant heterogeneity in Table 3 do not suffer from the the same unusual distributions and/or data cleaning procedures as do the four anchoring questions. But, because they are quite atypical exemplars of psychology experiments I would personally not base estimates of heterogeneity in psychology experiments in general from them. One is not an experiment, two involve items that are more culturally dependent than most psychological constructs: reactions to George Washington and an anti-democracy speech. The fourth is barely significantly heterogeneous: p=.04. But this is just my opinion [↩]For robustness I re-run the simulations keeping the number of observations in each condition constant within lab, the results were are just as bad. [↩]One could use these distributions to compute an adjusted p-value for heterogeneity; that is, compute how often we get a p-value as low as the one obtained by Many Labs, or an I2 as high as they obtained, among the shuffled datasets. But I do not do that here because such calculations should build in the data-cleaning procedures, and code for such data cleaning is not available. [↩]In particular, I made the effect size be proportional to the size of the name of the lab. So for the rows in the data where the lab it called “osu” the effect is d=.3, where it is “Ithaca” the effect is d=.6. As it happens, the average lab length is about 5 characters, so this works, and sd(length) is .18, which i round to .2 in the figure title.  [↩]BTW I noted that tau makes more sense conceptually than I2 . But mathematically, for a given dataset, tau is a simple transformation of it I2 (or rather, vice versa: I2=tau2/(total variance) ) thus if one is statistically invalid, so is the other. Tau is not a more robust statistic than I2 is. [↩]

[63] "Many Labs" Overestimated The Importance of Hidden Moderators

In a forthcoming Psych Science paper, Cuddy, Schultz, & Fosse, hereafter referred to as CSF, p-curved 55 power-posing studies (.pdf |  SSRN), concluding that they contain evidential value [1]. Thirty-four of those studies were previously selected and described as “all published tests” (p. 657) by Carney, Cuddy, & Yap (2015; .htm). Joe and Uri p-curved those 34 studies and concluded that they lacked evidential value (.htm | Colada[37]). The two p-curve analyses – Joe & Uri's old p-curve and CSF's new p-curve – arrive at different conclusions not because the different sets of authors used different sets of tools, but rather because they used the same tool to analyze different sets of data.In this post we discuss CSF's decision to include four studies with unusually small p-values (e.g., p < 1 in a quadrillion) in their analysis. The inclusion of these studies was sufficiently problematic that we stopped further evaluating their p-curve. [2].Aside:   Several papers have replicated the effect of power posing on feelings of power and, as Joe and Uri reported in their Psych Science paper (.htm, pg.4), a p-curve of those feelings-of-power effects suggests they contain evidential value. CSF interpret this as a confirmation of the central power-posing hypothesis, whereas we are reluctant to interpret it as such for reasons that are both psychological and statistical. Fleshing out the arguments on both sides may be interesting, but it is not the topic of this post.Evaluating p-curves Evaluating any paper is time consuming and difficult. Evaluating a p-curve paper – which is in essence, a bundle of other papers – is necessarily more time consuming and more difficult.We have, over time, found ways to do it more efficiently. We begin by preliminarily assessing three criteria. If the p-curve fails any of these criteria, we conclude that it is invalid and stop evaluating it. If the p-curve passes all three criteria, we evaluate the p-curve work more thoroughly.Criterion 1: Study Selection Rule Our first step is to verify that the authors followed a clear and reproducible study selection rule. CSF did not. That’s a problem, but it is not the focus of this post. Interested readers can check out this footnote: [3].Criterion 2: Test Selection Figure 4 (.pdf) in our first p-curve paper (SSRN) explains and summarizes which tests to select from the most common study designs. The second thing we do when evaluating a p-curve paper is to verify that the guidelines were followed by focusing on the subset of designs that are most commonly incorrectly treated by p-curvers. For example, we look at interaction hypotheses to make sure that the right test is included, and we look to see whether omnibus tests are selected (they should almost never be; see Colada[60]). CSF selected some incorrect test results (e.g., their smallest p-value comes from an omnibus test). See "Outlier 1" below.Criterion 3. Outliers Next we sort studies by p-value to identify possible outliers, and we carefully read the papers containing an outlier result. We do this both because outliers exert a disproportionate effect on the results of p-curve, and because outliers are much more likely to represent the erroneous inclusion of a study or the erroneous selection of a test result. This post focuses on outliers.This figure presents the distribution of p-values in CSF’s p-curve analysis (see their disclosure table .xlsx). As you can see, there are four outliers:Outlier 1 CSF's smallest p-value is from F(7, 140) = 19.47, approximately p = .00000000000000002, or 1 in 141 quadrillion. It comes from a 1993 experiment published in the journal The Arts in Psychotherapy (.htm).In this within-subject study (N = 24), each participant held three “open” and three “closed” body poses. At the beginning of the study, and then again after every pose, they rated themselves on eight emotions. The descriptions of the analyses are insufficiently clear to us (and to colleagues we sent the paper to), but as far as we can tell, the following things are true:(1) Some effects are implausibly large. For example, Figure 1 in their paper (.htm) suggests that the average change in happiness for those adopting the “closed” postures was ~24 points on a 0-24 scale. This could occur only if every participant was maximally happy at baseline and then maximally miserable after adopting every one of the 3 closed postures.(2) The statistical analyses incorrectly treat multiple answers by the same participants as independent, across emotions and across poses.(3) The critical test of an interaction between emotion valence and pose is not reported. Instead the authors report only an omnibus interaction: F(7, 140) = 19.47. Given the degrees-of-freedom of the test, we couldn’t figure out what hypothesis this analysis was testing, but regardless, no omnibus test examines the directional hypothesis of interest. Thus, it should not be included in a p-curve analysis.Outlier 2 CSF's second smallest p-value is from F(1,58)=85.9,  p = .00000000005, or 1 in 2 trillion. It comes from a 2016 study published in Biofeedback Magazine (.htm). In that study, 33 physical therapists took turns in dyads, with one of them (the “tester”) pressing down on the other’s arm, and the other (the “subject”) attempting to resist that pressure.The p-value selected by CSF compares subjective arm strength when the subject is standing straight (with back support) vs. slouching (without support). As the authors of the original article explain, however, that has nothing to do with any psychological consequences of power posing, but rather, with its mechanical consequences. In their words: “Obviously, the loss of strength relates to the change in the shoulder/body biomechanics and affects muscle activation recorded from the trapezius and medial and anterior deltoid when the person resists the downward applied pressure” (p. 68-69; emphasis added) [4].Outlier 3 CSF's third smallest p-value is from F(1,68)=26.25, p = .00000267, or 1 in ~370,000. It comes from a 2014 study published in Psychology of Women Quarterly (.htm). This paper explores two main hypotheses, one that is quite nonintuitive, and one that is fairly straightforward. The nonintuitive hypothesis predicts, among other things, that women who power pose while sitting on a throne will attempt more math problems when they are wearing a sweatshirt but fewer math problems when they are wearing a tank-top; the prediction is different for women sitting in a child's chair instead of a throne [5].CSF chose the p-value for the straightforward hypothesis, the prediction that people experience fewer positive emotions while slouching (“allowing your rib cage to drop and your shoulders to rotate forward”) than while sitting upright (“lifting your rib cage forward and pull[ing] your shoulders slightly backwards”).Unlike the previous two outliers, one might be able to validly include this p-value in p-curve. But we have reservations, both about the inclusion of this study, and about the inclusion of this p-value.First, we believe most people find power posing interesting because it affects what happens after posing, not what happens while posing. For example, in our opinion, the fact that slouching is more uncomfortable than sitting upright should not be taken as evidence for the power poses hypothesis.Second, while the hypothesis is about mood, this study’s dependent variable is a principal component that combines mood with various other theoretically irrelevant variables that could be driving the effect, such as how "relaxed" or "amused" the participants were. We discuss two additional reservations in this footnote: [6].Outlier 4 CSF's fourth smallest p-value is from F(2,44)=13.689, p=.0000238, or 1 in 42,000. It comes from a 2015 study published in the Mediterranean Journal of Social Sciences (.htm). Fifteen male Iranian students were all asked to hold the same pose for almost the entirety of each of nine 90-minute English instruction sessions, varying across sessions whether it was an open, ordinary, or closed pose. Although the entire class was holding the same position at the same time, and evaluating their emotions at the same time, and in front of all other students, the data were analyzed as if all observations were independent, artificially reducing the p-value.Conclusion Given how difficult and time consuming it is to thoroughly review a p-curve analysis or any meta-analysis (e.g., we spent hours evaluating each of the four studies discussed here), we preliminarily rely on three criteria to decide whether a more exhaustive evaluation is even warranted. CSF's p-curve analysis did not satisfy any of the criteria. In its current form, their analysis should not be used as evidence for the effects of power posing, but perhaps a future revision might be informative.Author feedback. Our policy (.htm) is to share, prior to publication, drafts of posts with original authors whose work we discuss, asking them to identify anything that is unfair, inaccurate, misleading, snarky, or poorly worded. We contacted CSF and the authors of the four studies we reviewed. Amy Cuddy responded to our email, but did not discuss any of the specific points we made in our post, or ask us to make any specific changes. Erik Peper, lead author of the second outlier study, helpfully noticed that we had the wrong publication date and briefly mentioned several additional articles of his own on how slouched positions affect emotions, memory, and energy levels (.htm; .pdf; htm; html; html). We also received an email from the second author of the first outlier study; he had "no recommended changes." He suggested that we try to contact the lead author but we were unable to find her current email address. Subscribe to Blog via EmailEnter your email address to subscribe to this blog and receive notifications of new posts by email.  Email Address        Subscribe  FootnotesWhen p-curve concludes that there is evidential value, it is simply saying that at least one of the analyzed findings was unlikely to have arisen from the mere combination of random noise and selective reporting. In other words, at least one of the studies would be expected to repeatedly replicate [↩]After reporting the overall p-curve, CSF also split the 55 studies based on the type of dependent variable: (i) feelings of power, (ii) EASE (“Emotion, Affect, and Self-Evaluation”), and (iii) behavior or hormonal response (non-EASE). They find evidential value for the first two, but not the last. The p-curve for EASE includes all four of the studies described in this post. [↩]To ensure that studies are not selected in a biased manner, and more generally to help readers and reviewers detect possible errors, the set of studies included in p-curve must be determined by a predetermined rule. The rule, in turn, should be concrete and precise enough that an independent set of researchers following the rule would generate the same, or virtually the same, set of studies. The rule, as described in CSF's paper, lacks the requisite concreteness and precision. In particular, the paper lists 24 search terms (e.g., “power”, “dominance”) that were combined (but the combinations are not listed). The resulting hits were then “filter[ed] out based on title, then abstract, and then the study description in the full text” in an unspecified manner. (Supplement: https://osf.io/5xjav/ | our archived copy .txt). In sum, though the authors provide some information about how they generated their set of studies, neither the search queries nor the filters are specified precisely enough for someone else to reproduce them. Joe and Uri's p-curve, on the other hand, followed a reproducible study selection rule: all studies that were cited by Carney et al. (2015) as evidence for power posing. [↩]The paper also reports that upright vs. collapsed postures may affect emotions and the valence of memories, but these claims are supported by quotations rather than by statistics. The one potential exception is that the authors report a “negative correlation between perceived strength and severity of depression (r=-.4)." Given the sample size of the study, this is indicative of a p-value in the .03-.05 range. The critical effect of pose on feelings, however, is not reported. [↩]The study (N = 80) employed a fully between-subjects 2(self-objectification: wearing a tanktop vs. wearing a sweatshirt) x 2(power/status: sitting in a “grandiose, carved wooden decorative antique throne” vs. a “small wooden child’s chair from the campus day-care facility”) x 2(pose: upright vs slumped) design. [↩]First, for robustness, one would need to include in the p-curve the impact of posing on negative mood, which is also reported in the paper and which has a considerably larger p-value (F = 13.76 instead of 26.25). Second, the structure of the experiment is very complex, involving a three-way interaction which in turn hinges on a two-way reversing interaction and a two-way attenuated interaction. It is hard to know if the p-value distribution of the main effect is expected to be uniform under the null (a requirement of p-curve analysis) when the researcher is interested in these trickle-down effects. For example, it hard to know whether p-hacking the attenuated interaction effect would cause the p-value associated with the main effect to be biased downwards. [↩]

Read Data Colada

Choose Your Reading Path

Full Backlog

Best Hits

You'll start with:

[1] "Just Posting It" works, leads to new retraction in Psychology

[2] Using Personal Listening Habits to Identify Personal Music Preferences

[3] A New Way To Increase Charitable Donations: Does It Replicate?

About Data Colada