Does well-designed, well-documented, psychologically principled, and carefully implemented psychotherapy lead to better outcomes than therapy of lower quality?

Jones, Payton J., Patrick Mair, Sofie Kuppens, and John R. Weisz. 2019. “An Upper Limit to Youth Psychotherapy Benefit? A Meta-analytic Copula Approach to Psychotherapy Outcomes.” OSF Preprints. March 28. doi:10.31219/osf.io/jsmf5

Abstract: Across 50 years of research, extensive efforts have been made to improve the effectiveness of psychotherapies for children and adolescents. Yet recent evidence shows no significant improvement in youth psychotherapy outcomes. In other words, efforts to improve the general quality of therapy models do not appear to have translated directly into improved outcomes. We used multilevel meta-analytic data from 502 randomized controlled trials to generate a bivariate copula model predicting effect size as therapy quality approaches infinity. Our results suggest that even with a therapy of perfect quality, achieved effect sizes may be modest. If therapy quality and therapy outcome share a correlation of 0.20 (a somewhat optimistic assumption given the evidence we review), a therapy of perfect quality would produce an effect size of Hedge's g = 0.83. We suggest that youth psychotherapy researchers complement their efforts to improve psychotherapy quality by investigating additional strategies for improving outcomes.

---
Does well-designed, well-documented, psychologically principled, and carefully implemented psychotherapy lead to better outcomes than therapy of lower quality? Empirical evidence on the association between therapy "quality" and therapy outcome is more mixed than one might expect.
 The literature reveals varying opinions on what constitutes a therapy of high quality. One view suggests that an important dimension of therapy quality is the presence of advantageous "specific factors" or "theory-specified factors" in psychotherapies (Castonguay & Grosse, 2005; Webb, DeRubeis, & Barber, 2010). These researchers emphasize the idea that the content of therapy is an important dimension of therapy quality. They stress the frequency with which certain theoretically-driven approaches involving specific content, such as exposure and response prevention for OCD, outperform other psychotherapies (DeRubeis, Brotman, & Gibbons, 2005). The "theory-specified factors"
perspective implies that randomized-controlled trials comparing different types of therapy are of great importance to the scientific literature, as this approach is likely to reveal the types of therapeutic content that are most effective in reducing psychopathology.
While differing specific factors and treatment types have often dominated the discussion of therapy, some influential theorists and researchers have discounted the importance of these factors.
Since 1936, some influential figures in the field have argued that the specific steps followed in therapy may have little impact, relative to the influence of certain common factors (Messer & Wampold, 2002; Rosenzweig, 1936). One prominent version of this perspective has been labeled the "Dodo Bird" conjecture, in reference to the character in Lewis Carol's book Alice in Wonderland, who proclaims: "Everybody has won, and all must have prizes." The Dodo Bird conjecture proposes that diverse types of therapies are equally effective, provided that they possess certain common factors. One aspect of this perspective is the notion that, across a broad range of “bona fide” therapies, the specific factors associated with therapy quality bear little relation to therapy outcome (Wampold et. al, 1997).
Importantly, the common factors approach does not discount the notion that therapy quality matters – it simply emphasizes different types of therapy quality which are independent of therapeutic content, such as adequate therapist training in facilitative interpersonal skills (Anderson, McClintock, Himawan, Song, & Patterson, 2016).
The Dodo Bird hypothesis has generated both controversy and data synthesis, with various meta analyses and reviews cited in support of each position in the debate. Some meta-analyses have failed to identify therapy type as a significant moderator of outcome (e.g., Baardseth et al., 2013; Miller, Wampold, & Verhely, 2008; Wampold et al., 1997), and these have been cited as support for the Dodo
Bird conjecture. Findings of other meta-analyses and some reviews have indicated that the specific procedures performed in therapy matter. For instance, some of these have identified significant between therapy type differences in magnitude of effect size for various treated problems (e.g., Chambless & Ollendick, 2001; Hunsley & DeGuilio, 2002; Weiss & Weisz, 1995; Weisz, Weiss, Han, Granger, &
Morton,1995), and others have identified therapies for which evidence shows adverse effects (Lilienfeld, 2007). The existence of harmful therapies suggests that the dimension of therapy quality is related to therapy outcome, at least on the extreme low end of therapy quality (e.g., where therapy is designed in
opposition to psychological principles). In other relevant work, researchers have shown that type of therapy can have a marked impact when symptoms are especially severe (e.g., Lorenzo-Luaces, DeRubeis, van Straten, & Tiemens, 2017). As these examples indicate, the various syntheses of evidence have suggested that therapy content may matter in some cases but not in all. Both theoretical camps promote the idea that psychotherapy varies in its quality and its quality can be improved, but the camps differ as to how this should be done. Researchers in the specific factors camp have focused on improving therapy content, whereas those in the common factors camp have focused on maximizing factors that exist independent of therapy content, such as therapist skills in interacting with clients.
Both perspectives are relevant to the present paper, in which we focus on the relation between psychotherapy quality and psychotherapy outcome. To define psychotherapy quality in a way that encompasses both perspectives, we have tried to synthesize points from both sides of the debate. The specific factors view suggests that high-quality in therapy will include the use of procedures that have a basis in sound psychological principles and the accumulation of evidence from empirical studies, together with training of therapists in the specific procedures involved, and insuring adherence to the specified protocols (e.g., Chambless & Ollendick, 2001). The common factors view suggests that high quality in therapy will include an array of therapist characteristics, such as skill in the interpersonal
aspects of working with clients. For purposes of the present paper, we include both perspectives, operationally defining "quality of therapy" to include both (1) the specific contents of therapy protocols and the procedures (e.g., therapist training) used to ensure faithful delivery of those contents, and (2) common factors (e.g., therapist interpersonal skills) that may influence the conduct of therapy independently of specific treatment content, provided that the procedures or elements of 1 and 2 are intended to improve client outcome and do not depend upon client factors.
To better understand how quality is defined and operationalized throughout this paper, one can use the metaphor of a psychological scale that measures therapy quality. Our metaphorical scale for quality would include a list of items that derive from both the specific factors and the common factors approach. When we refer to quality as a general principle, we refer to the metaphorical sum-score of all
items on the scale (or perhaps more precisely as an extracted principal component from all items). That is, we aim to represent quality as an abstract dimension comprising all relevant aspects of high-quality therapies.


How much does quality matter?

As noted, researchers have argued over what constitutes a high-quality therapy. But to what degree do these different aspects of therapy quality predict outcome? We reviewed the literature on psychotherapy quality in an exploratory search. Our aim was to identify empirical articles which estimated the relationship between therapy outcome and some aspect of psychotherapy quality. Because research in this area is scarce, we broadened our review of this area to include both adult and youth
focused therapies.
To make sure we had adequately represented diverse views on this issue, we contacted prominent psychotherapy researchers and asked them to recommend studies. To identify researchers, we searched PSYCinfo for the period from January 1990 to January 2018 using the terms "common factors in psychotherapy," "empirically supported psychotherapy," and "psychotherapy quality." We identified
authors of the identified publications who were frequently cited for work related to these topics. In addition, we identified current and past editors of journals in which psychotherapy research is frequently published. The resulting list of authors included 19 prominent researchers with diverse theoretical perspectives: David Barlow, Larry Beutler, Ronald Brown, Dianne Chambless, David Clark, Michelle
Craske, Joanne Davila, Robert DeRubeis, Judy Garber, Mark Hilsenroth, Steven Hollon, Alan Kazdin, Philip Kendall, Michael Lambert, John Norcross, Francheska Pereplechikova, Dan Strunk, and Bruce Wampold. We sent an email 1 to each author requesting that they identify as the most scientifically sound study in which the relationship between quality and outcome was assessed.

 1 "Dear Professor ________:
As part of a new study examining the relation between psychotherapy quality and psychotherapy outcome, we are contacting you and a small number of other prominent psychotherapy researchers. We ask you to please identify what you regard as the most scientifically sound study in the literature in which investigators assessed the relation between a credible measure of psychotherapy quality, on the one hand, and therapy outcome, on the other hand.

 If you can provide either a PDF or an exact reference for the study you identify, we will be very grateful. Thanks very much for considering our request."

 Twelve of the nineteen authors responded to our request. Some of the authors declined to provide a study, noting theoretical concerns with the idea of therapy quality, or concerns related to unfamiliarity with more recent literature on therapy outcomes. Other authors provided more than one study. All studies provided, including results from our own initial literature review, were initially considered as part of the exploratory analysis.
After reviewing the full pool of nominated studies, we excluded several studies from further analyses due to 1) failure to report effect sizes 2) failure to include a discernable measure of therapy quality or 3) use of therapy quality measures that depended, in full or in part, upon client factors (e.g., therapeutic alliance between therapist and client). A list of excluded studies and reasons for exclusion can be found in the supplemental materials.
The results of our exploratory search are presented in Table 1. We measured a variety of different types of therapy quality, ranging from treatment type to therapist competence to therapist facilitative interpersonal skills. Our review included both single studies of therapy quality as well as meta-analyses of evidence across many studies. Aside from comparing different types of psychotherapy, none of the studies in our search included experimental manipulations of therapy quality, indicating an important area of research that may be neglected.
A histogram of the pooled effect sizes is presented in Figure 1. A high bar in this histogram indicates that many effect sizes in the literature fell within the range of effect indicated on the x-axis.
For instance, the first bar in the graph indicates that 17 effect sizes included in our review fell in the range between 0.00 and 0.02. Effect sizes are given as r2 type, which reflects the proportion of variance accounted for by therapy quality on therapy outcome (see Fritz, Morris, & Richler, 2012). In summary, most effect sizes were close to zero, indicating that higher therapy quality did not relate to better therapy
outcome. Meta-analyses and more recent studies in general reported smaller effects compared to individual studies and older studies. All meta-analytic effects fell between 0 and 0.005. In general, these exploratory analyses showed an association between therapy quality and therapy outcome that was quite modest, at best.

How good can therapy be? A focus on youth psychotherapy

After synthesizing findings of the studies listed in Table 1, we were interested in applying what could be learned from that synthesis to estimate the extent to which improving therapy quality might improve psychotherapy outcome. For that purpose, we needed a large pool of psychotherapy outcome studies. Because therapy procedures and protocols, as well as required therapist skills, are quite different
for treatment of children and adolescents (herein “youths”) than adults, we thought it best to focus on one or the other age group. While both age groups are important, our past research on youth psychopathology and psychotherapy, plus the fact that we had access to data from 502 randomized controlled trials of youth psychotherapy, led to our focus on therapy with young people. This work
illustrates a procedure that could, of course, be applied in the future to any group, defined by age or any other factor. Using this large youth psychotherapy dataset, we sought to determine the efficacy of an "optimal quality" therapy – that is, a therapy in which all beneficial clinician factors were maximized. In other words, we were not interested in answering the question of "how good is therapy", but rather in answering the question of "how good could therapy be?" Answering this question is akin to an optimization problem in mathematics. First, a function must be specified which describes relevant inputs (therapy quality) and outputs (therapy outcome). The function is then analyzed to identify the point at which a maximal output is achieved based on the inputs. We sought to answer this question based on relevant knowledge regarding youth psychotherapy quality and outcome.

To address these questions using empirical data, we first generated a bivariate distribution function, known as a copula, between therapy quality and treatment outcome drawing upon an extensive meta-analysis of randomized controlled trials (RCTs) of youth psychotherapy. We then utilized our simulated distribution to predict the upper limit of effect size as therapy quality approaches infinity. In other words, we posed the question: if we could design a youth psychotherapy of perfect quality, how effective would it be?