Psychology's Renaissance. Leif D. Nelson, Joseph P. Simmons, and Uri Simonsohn. Annual Review of Psychology, forthcoming. https://doi.org/10.1146/annurev-psych-122216-011836
Abstract: In 2010–2012, a few largely coincidental events led experimental psychologists to realize that their approach to collecting, analyzing, and reporting data made it too easy to publish false-positive findings. This sparked a period of methodological reflection that we review here and call Psychology’s Renaissance. We begin by describing how psychologists’ concerns with publication bias shifted from worrying about file-drawered studies to worrying about p-hacked analyses. We then review the methodological changes that psychologists have proposed and, in some cases, embraced. In describing how the renaissance has unfolded, we attempt to describe different points of view fairly but not neutrally, so as to identify the most promising paths forward. In so doing, we champion disclosure and preregistration, express skepticism about most statistical solutions to publication bias, take positions on the analysis and interpretation of replication failures, and contend that meta-analytical thinking increases the prevalence of false positives. Our general thesis is that the scientific practices of experimental psychologists have improved dramatically.
Keywords: p-hacking, publication bias, renaissance, methodology, false positives, preregistration
---
Psychologists have long been aware of two seemingly contradictory problems with the published literature. On the one hand, the overwhelming majority of published findings are statistically significant (Fanelli 2012, Greenwald 1975, Sterling 1959). On the other hand, the overwhelming majority of published studies are underpowered and, thus, theoretically unlikely to obtain results that are statistically significant (Chase & Chase 1976, Cohen 1962, Sedlmeier & Gigerenzer 1989). The sample sizes of experiments meant that most studies should have been failing, but the published record suggested almost uniform success.
There is an old, popular, and simple explanation for this paradox. Experiments that work are sent to a journal, whereas experiments that fail are sent to the file drawer (Rosenthal 1979). We believe that this “file-drawer explanation” is incorrect. Most failed studies are not missing. They are published in our journals, masquerading as successes.
The file-drawer explanation becomes transparently implausible once its assumptions are made explicit. It assumes that researchers conduct a study and perform one (predetermined) statistical analysis. If the analysis is significant, then they publish it. If it is not significant, then the researcher gives up and starts over. This is not a realistic depiction of researcher behavior. Researchers would not so quickly give up on their chances for publication, nor would they abandon the beliefs that led them to run the study, just because the first analysis they ran was not statistically significant. They would instead explore the data further, examining, for example, whether outliers were interfering with the effect, whether the effect was significant within a subset of participants or trials, or whether it emerged when the dependent variable was coded differently. Pre-2011 researchers did occasionally file-drawer a study, although they did not do so when the study failed, but rather when p-hacking did. Thus, whereas our file drawers are sprinkled with failed studies that we did not publish, they are overflowing with failed analyses of the studies that we did publish.
34 Inspiring Quotes on Criticism (and How to Handle It) -- EXTRACT
- – Dale Carnegie
- “The pleasure of criticizing takes away from us the pleasure of being moved by some very fine things.”
– Jean de La Bruyère - – Aristotle
- – John Wooden
- “Criticism is an indirect form of self-boasting.”
– Emmet Fox - “When virtues are pointed out first, flaws seem less insurmountable.”
– Judith Martin - – Neil Gaiman
- – Norman Vincent Peale
- “When we judge or criticize another person, it says nothing about
that person; it merely says something about our own need to be
critical.”
– Unknown - “It is much more valuable to look for the strength in others. You can gain nothing by criticizing their imperfections.”
– Daisaku Ikeda - “The artist doesn’t have time to listen to the critics. The ones who
want to be writers read the reviews, the ones who want to write don’t
have the time to read reviews.”
– William Faulkner - “If we judge ourselves only by our aspirations and everyone else
only their conduct we shall soon reach a very false conclusion.”
– Calvin Coolidge - “I have yet to find the man, however exalted his station, who did
not do better work and put forth greater effort under a spirit of
approval than under a spirit of criticism.”
– Charles Schwab - “I criticize by creation, not by finding fault.”
– Marcus Tullius Cicero - “Let me never fall into the vulgar mistake of dreaming that I am persecuted whenever I am contradicted.”
– Ralph Waldo Emerson - “Don’t criticize what you don’t understand, son. You never walked in that man’s shoes.”
– Elvis Presley - – Frank A. Clark
- “People tend to criticize their spouse most loudly in the area where they themselves have the deepest emotional need.”
– Gary Chapman - “Criticism is the disapproval of people, not for having faults, but having faults different from your own.”
– Unknown - “It is not the critic who counts; not the man who points out how the
strong man stumbles, or where the doer of deeds could have done them
better.
The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly; who errs and comes short again and again; because there is not effort without error and shortcomings; but who does actually strive to do the deed; who knows the great enthusiasm, the great devotion, who spends himself in a worthy cause, who at the best knows in the end the triumph of high achievement and who at the worst, if he fails, at least he fails while daring greatly.
So that his place shall never be with those cold and timid souls who know neither victory nor defeat.”
– Theodore Roosevelt - “Before you go and criticize the younger generation, just remember who raised them.”
– Unknown - “Who do you spend time with? Criticizers or encouragers? Surround
yourself with those who believe in you. Your life is too important for
anything less.”
– Steve Goodier - “Criticism may not be agreeable, but it is necessary. It fulfills
the same function as pain in the human body. It calls attention to an
unhealthy state of things.”
– Winston Churchill - “He has a right to criticize, who has a heart to help.”
– Abraham Lincoln - – Antoine de Saint-Exupery
- – Eleanor Roosevelt
- “One mustn’t criticize other people on grounds where he can’t stand perpendicular himself”
– Mark Twain - “That was excellently observed’, say I, when I read a passage in an
author, where his opinion agrees with mine. When we differ, there I
pronounce him to be mistaken.”
– Jonathan Swift - “Any fool can criticize, condemn and complain and most fools do.”
– Benjamin Franklin - “Flatter me, and I may not believe you. Criticize me, and I may not
like you. Ignore me, and I may not forgive you. Encourage me, and I will
not forget you. Love me and I may be forced to love you.”
– William Arthur Ward - “A man interrupted one of the Buddha’s lectures with a flood of abuse.
Buddha waited until he had finished and then asked him:
If a man offered a gift to another but the gift was declined, to whom would the gift belong?
To the one who offered it, said the man.
Then, said the Buddha, I decline to accept your abuse and request you to keep it for yourself.” - – Joseph Joubert
- – Abraham Lincoln
- – Michel de Montaigne