Selection Bias and the Statistical Analysis of Supreme Court Decision Making
Understandably, several investigations of whether the Roberts Court is “business friendly” have used statistical analysis to help answer the question. This general approach—using large sets of decisions, coded on a particular dependent variable—is often essential for uncovering broad patterns in judicial behavior. Of course, there are limitations to what this sort of analysis can reveal. Most prominently, these studies typically ignore the conscious objectives articulated by the justices in their written opinions. Instead, they focus exclusively on the outcomes as they relate to a particular variable (such as whether it favors or harms civil rights plaintiffs). But that variable may not be salient to the justices in many of the studied decisions. Consider, for example, the Roberts Court’s business decisions. On several occasions, the business aspects of a case have hardly seemed relevant to the justices. In Free Enterprise v. PCAOB,12 for instance, the majority seemed driven by its conception of Article II and the separation of powers in analyzing the removal limitations on Public Company Accounting Oversight Board members, not the impact of the Sarbanes-Oxley Act on private companies. And in Chamber of Commerce v. Whiting13—in which Chief Justice Roberts and Justices Scalia, Kennedy, Thomas, and Alito voted against the Chamber, and Justices Ginsburg, Breyer, and Sotomayor voted in its favor—the justices appeared to be moved much more by their feelings about illegal immigration (and perhaps federalism) than by the impact of the Arizona Legal Workers Act on business activity. From a statistical perspective, though, Free Enterprise was simply a decision favoring business interests, and Whiting a decision against them.
By design, then, outcome-based statistical analysis is oblivious to what the justices consciously seek to accomplish—what they care about. But so long as we understand what these studies aim to measure, this limitation may be more of a feature than a bug. The point of statistical analysis of Supreme Court decision making is to explore whether those decisions systematically point in particular directions, regardless of the justices’ subjective motivations or articulated rationales. By coding decisions based on objectively verifiable characteristics, these investigations can uncover important patterns that those immersed in the Court’s work—such as lawyers, commentators, or scholars—are susceptible to overlook.
It is also understandable that studies examining the Roberts Court’s receptiveness to the claims of business litigants have often used the participation of the Chamber of Commerce’s litigation office, the National Chamber Litigation Center (NCLC), as a proxy for those interests. To be sure, the Chamber’s participation is an underinclusive signal; the Roberts Court has handed down several decisions important to American business in which the Chamber has not filed a brief.14 One reason is that, because the Chamber appears regularly before particularly true when plausible arguments support one’s position, as is invariably the case for the types of issues the Supreme Court decides." Id. at 433 (citations omitted).
- 12 130 S. Ct. 3138 (2010).
- 13 131 S. Ct. 1968 (2011).
- 14 There are several examples from October Term 2010 alone. See, e.g., Costco Wholesale Corp. v. Omega S.A., 131 S. Ct. 565 (2010) (per curiam) (addressing whether the first-sale doctrine in copyright law applies to goods manufactured outside the United States); Kasten v. Saint-Gobain Performance Plastics Corp., 131 S. Ct. 1325 (2011) (holding that complaints “filed" under the antiretaliation provision of the Fair Labor Standards Act can be made orally, as well as in writing); Pliva, the Court, it has strategic reasons to cultivate and preserve its credibility with the justices, steering it toward filing briefs only when it can present fairly strong arguments (regardless of a case’s substantive importance to business interests). Another reason is that the Chamber has a policy of not participating in cases pitting business litigants against one another, even if one outcome might favor business interests writ large.15 On balance, though, the Chamber’s participation is a fair indication that the case is significant to the business community as a whole; most Supreme Court decisions have some ramifications for private enterprise, but the Chamber’s involvement suggests that the implications are more than de minimis.16 It is also a decent signal that one outcome in the case decidedly favors business interests, at least in general terms.
In fact, defining business-related cases based on the Chamber’s participation may be superior to approaches used in some more sophisticated statistical analyses, which have cast the net more broadly. For example, in their study of the Court’s business decisions for the New York Times, Lee Epstein, William Landes, and Richard Posner included, inter alia, every case categorized under the heading “Economic Activity” in the Supreme Court Database.17 This category includes decisions addressing, among other things, intellectual property law (patent, trademark, and copyright), regulated industries (natural gas, electric power, and telecommunications), and antitrust law.18 Including such cases may be perfectly appropriate in assessing the number of business-related decisions the Court has rendered over time. But as Epstein, Landes, and Posner have since explained,19 using this category to analyze whether the Court’s decisions have favored business interests creates problems. In each of these areas of law, different firms typically have competing interests on each side of a given question. For instance, the Court’s recent decision in Microsoft Corp. v. i4i Limited Partnership20 benefitted businesses presently holding patents, as it adopted a higher standard of proof for an alleged infringer to establish a patent’s invalidity as a defense. But the outcome hurt corporations like Microsoft that, at least sometimes, engage in potentially infringing activity. In cases like this—absent a more sophisticated, finely
Inc. v. Mensing, 131 S. Ct. 817 (2010) (addressing whether federal law preempts state tort failure-to- warn claims against the generic manufacturers of drugs, where the generic’s label is identical to the label of the branded drug).
- 15 See Franklin, supra note 1, at 1023.
- 16 See Epstein, Landes & Posner, supra note 3, at 1439.
- 17 See Epstein et al., supra note 10, at 1 (referencing http://supremecourtdatabase.org/ (last accessed February 18, 2016)).
- 18 See Supreme Court Database Online Code Book, available at http://supremecourtdatabase. org/documentation.php?var=issue (last accessed February 18, 2016).
- 19 See Epstein, Landes & Posner, How Business Fares, supra note 10, at 1437-45.
- 20 No. 10-290, 2011 WL 2224428 (U.S. June 9, 2011).
grained evaluation of the holding—neither outcome is properly identifiable as probusiness.   Using the Chamber’s participation to define the relevant universe (and the outcome that favors business interests) avoids this problem; it singles out those decisions in which the business community, as a whole, stood to gain from one result or the other.
Thus, neither the focus on outcomes, nor the use of the Chamber of Commerce as a proxy for business interests, is especially troublesome in a statistical analysis ofwhether the Roberts Court has been probusiness. More problematic is that these studies have typically failed to account for the possibility of selection bias in the issues the Court has decided on the merits. Put simply, the questions on which the Court grants certiorari are not a random draw of disputed issues of federal law. Rather, they are shaped by forces that can bias that universe in several ways.
A particularly potent such force is the content of the Court’s prior decisions. Consider the Court’s two recent decisions involving the problem of global warming. In 2007, the Supreme Court held in Massachusetts v. EPA22 that greenhouse gases (such as carbon dioxide) constitute “air pollutants” under the Clean Air Act, such that the Environmental Protection Agency (EPA) has jurisdiction to regulate their emission across the entire US economy. For obvious reasons, this decision was hugely important—perhaps the Roberts Court’s most significant business-related decision to date. In 2011, however, the Court ruled against a coalition of state governments and environmental interests in a second global warming case, American Electric Power Co. v. Connecticut.23 Specifically, the Court held that the Clean Air Act (and the regulatory authority it grants the EPA) displaces any federal common law on the subject of greenhouse gas emissions, thereby precluding the plaintiffs’ federal common-law interstate nuisance claim.
An outcome-based statistical analysis would suggest that the Court has been “business neutral” in its global warming dyad, with one decision favoring business interests and one against. But this misses the critical fact that Massachusetts v. EPA moved the universe of relevant legal questions in a proregulatory, antibusiness direction. Indeed, the precise question on which the Court granted certiorari in American Electric Power was “[w]hether a cause of action to cap carbon dioxide emissions can be implied under federal common law where no statute creates such a cause of action, and the Clean Air Act speaks directly to the same subject matter and assigns federal responsibility for regulating such emissions to the [EPA].”25 Massachusetts v. EPA was taken as a given, and what remained open was the comparatively marginal question of the federal judiciary’s leeway to supplement the EPA’s authority with interstate nuisance law. Business interests prevailed on this second question, but the two decisions together moved the law in a decidedly proregulatory direction.
We can conceptualize this phenomenon spatially, along a spectrum from the most proregulatory to the most probusiness state of the law (see Figure 1.1). (Again, compressing the content of Supreme Court decisions into two dimensions is an oversimplification, but it adequately illustrates the point here.) In essence, Massachusetts v. EPA selected M over M,. As a result, M became the center of gravity for the Court’s decision in American Electric Power such that, as the Court faced a choice between Ar and Ab a legal rule proximate to Mb was outside the realm of possibility. To be sure, the outcome in American Electric Power was favorable to business (moving the state of the law to Ab). But because the question on which the Court had granted certiorari was anchored by Mr, the sum of the two decisions moved the law in a proregulatory direction. Massachusetts v. EPA framed—and thus skewed—the universe of questions for the Court to decide subsequently.
Or consider the impact of Congress’s recent enactment of an important, far- reaching statute imposing a number of new obligations on business entities, a statute like the Americans with Disabilities Act (ADA). Given the broad range of rights conferred (and responsibilities imposed) by the ADA, the act’s precise contours have been the subject of frequent appellate litigation; indeed, the Supreme Court has averaged roughly one ADA case per term since the statute’s passage in 1990. In many of those cases, the Court has reached outcomes favoring business interests over disabled (or arguably disabled) individual plaintiffs. But if, in its substance, the ADA is largely an “antibusiness” statute, then the Court’s decisions might actually be “business neutral” in their content, even if a majority of the outcomes have favored ADA defendants. More broadly, the substance of statutes like the ADA—and the nature of their ambiguities, which then become the source of appellate litigation—inevitably shapes the Court’s agenda.
Or consider the potential distortions caused by the strategic behavior of litigants. As Richard Lazarus has documented, over the past twenty years corporations have increasingly hired lawyers with Supreme Court expertise for matters 
A spatial model illustrating how prior Supreme Court decisions can result in selection bias in the subsequent questions decided by the Court
As a result, they are more apt to press questions only when they believe the issues are truly cert worthy, and when their clients will ultimately win on the merits if the Court grants review. To be sure, experienced Supreme Court lawyers often represent employees, consumers, and investors at the Court, but this is more likely after the justices have set the case down for plenary review. At the certiorari stage—where the Court’s resources are the most stretched, such that counsel can be most influential—the specialized Supreme Court bar disproportionately represents the interests of the business community. All of this means that the business-related legal questions that the Court ultimately decides could well be ones on which business litigants are disproportionately likely to prevail.
These influences—the content of the Court’s prior decisions, Congress’s recent legislative activity, and the litigants’ strategic decisions—are just three of the factors that shape the pool of legal questions the Court resolves. There are several others. The larger point is that there is no clear way to determine whether the universe of issues that the Supreme Court ultimately decides is neutral with respect to the pertinent object of inquiry—as relevant here, the impact of the decisions on American business. As a result, no examination of a particular constituency’s “win rate” at the Court can, by itself, establish that the Court has favored that constituency. This is true even of quite careful and refined studies, such as that recently published by Epstein, Landes, and Posner in the Minnesota Law Review. They created two unique data sets that substantially improved on the “Economic Activity” category in the Supreme Court database, at least for purposes of measuring the business valence of the Court’s decisions. They then examined the voting records of every justice since the 1946 Term in those cases contained in the two data sets. As relevant here, Epstein, Landes, and Posner found that “the Roberts Court is much friendlier to business than” the Warren, Burger, or Rehnquist Courts. More specifically, five Roberts Court justices— Roberts, Scalia, Kennedy, Thomas, and Alito—rank among the top “ten Justices who, over the span of our study (the 1946 through 2011 Terms), have been the most favorable to business,” and Roberts and Alito “rank at the very top among the thirty-six Justices in our study.”
Again, the Epstein, Landes, and Posner investigation is extraordinarily careful, and it is much more sophisticated methodologically than the study I present in the following text. It plainly marks an important step toward uncovering the present Court’s inclinations in business-related cases. But it cannot conclusively establish that the Roberts Court has been friendlier to business interests than the Warren, Burger, or Rehnquist Courts unless we assume that the substantive content of the business-related questions the Court has resolved over this span has remained relatively constant. (This might be an entirely reasonable assumption, but it requires proof.) Furthermore, even ifwe so assume, the Epstein, Landes, and Posner study could only establish that the Roberts Court has been friendlier to business interests than its predecessor Courts, not that it has favored businesses relative to other litigants. The nub of the problem is that we cannot say whether the success rate of business interests demonstrates that the Roberts Court is truly probusiness without knowing how a “business neutral” Court would have ruled in precisely the same set of cases.
One way to solve this problem is to elide it—or, more specifically, to redefine the relevant question. Instead of examining a particular constituency’s success at the Court in isolation, we can compare that constituency’s success in the same universe of cases to another perennial litigant, one whose importance at the Court is relatively well established. And there is no better referent for these purposes than the Solicitor General, the officer who represents the federal government at the Supreme Court. As two political scientists recently explained, the Solicitor General “has long been presumed to enjoy a special relationship with the Supreme Court,” and “research has demonstrated over and over again that the U.S. government is far more successful than any other party or amicus curiae.” As Lazarus explains, “The Court plainly provides the Solicitor General’s legal arguments with heightened respect because of the nature of his client—the United States—and the deference that the judicial branch naturally owes in many legal settings to the views of counsel representing the interests of the two other branches of government.” More than that, though, attorneys in the Solicitor General’s Office “immerse themselves in the work of the Court,” and thus “become completely familiar with the Justices and their precedent, including their latest concerns and the inevitable cross-currents between otherwise seemingly unrelated cases that would be largely invisible to those who focus on just one case at a time"
To be sure, a comparative study is no panacea. It solves the selection bias problem because it compares the success of the relevant constituency and the referent litigant in precisely the same universe of decisions, such that the meaning of the findings does not depend (or depend so heavily) on the content of the questions presented. But in doing so, it necessarily introduces a new problem: The Court’s receptiveness to positions advocated by the referent litigant is unlikely to remain constant, undermining its usefulness as a benchmark. For instance, the success of the Solicitor General at the Court is apt to vary depending (among other things) on (l) the nature of the issue under review, (2) the composition of the Court, and (3) who occupies the White House. Thus, the core of the empirical problem remains unsolved: business’s rate of success at the Court—even when compared to that of the Solicitor General in the same universe of decisions—will not necessarily demonstrate whether the Court favors business litigants.
Nonetheless, comparing the success of business interests to that of the Solicitor General can be useful. First, the historical influence of the Solicitor General is well documented. Thus, even if the Court’s tendency to agree with the Solicitor General may wax and wane, the success of business relative to the federal government is a meaningful—if imprecise—indication of business’s success at the Court. Second, a comparison of the success rates of business litigants and the Solicitor General in business-related cases is interesting in its own right, especially in cases in which they have taken opposing positions. Currently, the Chamber of Commerce and the Solicitor General are perhaps the two most significant litigants to appear regularly at the Court. Thus, how they have fared relative to one another—and how their comparative success has changed over time—can tell us something important about the justices’ decision making, even if it does not directly tell us whether the Roberts Court is probusiness.
-  See, e.g., Epstein, Landes & Posner, supra note 3; Lee Epstein, William M. Landes & Richard A.Posner, Is the Roberts Court Pro-Business, December 17, 2010, available at http://epstein.wustl.edu/research/RobertsBusiness.pdf (last accessed February 18, 2016); Franklin, supra note 1; Srinivasan& Joondeph, supra note 9.
-  The classic attitudinalist studies of Supreme Court decision making—studies demonstrating an association between the justices’ preexisting political commitments and the outcomes forwhich they have voted—are those published by Jeffrey Segal and Harold Spaeth. See Jeffrey A.Segal & Harold J. Spaeth, The Supreme Court and the AttiTUDiNAL Model Revisited(2002). Segal and Spaeth are forthrightly agnostic about the justices’ subjective intentions, conceding that their voting patterns may be largely (or even entirely) the product of motivated reasoning—subconscious influences that steer their analyses toward preferred policy results. As Spaeth and Segalexplain, “classic social psychological findings demonstrate [that] the ability to convince oneself ofthe propriety of what one prefers to believe psychologically approximates the human reflex. This is
-  Interestingly, the Epstein, Landes, and Posner study for the New York Times would categorize decisions that are “procompetition” as “liberal,” and thus antibusiness. See Epstein et al., supranote 10, at 1 n.1. Thus, it presumably would code Microsoft’s loss in Microsoft Corp. v. i4i LimitedPartnership as unfavorable to business interests.
-  549 U.S. 497 (2007).
-  No. 10-174, 2011 WL 2437011 (U.S. June 20, 2011).
-  Id. at *4.
-  Petition for a Writ of Certiorari, American Electric Power Co. v. Connecticut, No. 10-174, at i(U.S. August 2, 2010).
-  Id.
-  See Epstein, Landes & Posner, How Business Fares, supra note 3.
-  See id.
-  Id. at 1471.
-  Id.
-  See Jonathan H. Adler, Business and the Roberts Court Revisited (Again), Volokh Conspiracy,May 6, 2013 (noting that “the methodology chosen by the study’s authors—determining whether a business interest won or lost in each case and then tallying up the decisions and individual justices’ votes—doesn’t account for the content of the studies or the doctrinal baseline”), available at http ://www.volokh.com/2013/05/06/business-and-the-roberts-court-revisited-again/ (last accessed February 18, 2016).Of course, asking that any study of Supreme Court decisions conclusively establish such a hypothesis maybe setting the bar too high.
-  Andrea McAtee & Kevin T. McGuire, Two Studies of Advocacy, Lawyers, Justices, and IssueSalience: When and How Do Legal Arguments Affect the U.S. Supreme Court, 41 Law & Soc’y Rev.259 (2007).
-  Lazarus, supra note 26, at 1493. See also Remarks Commemorating Celebration 55: The Women'sLeadership Summit, 32 Harv. J.L. & Gender 233, 237 (2009) (relating comments by JusticeGinsburg about the influence of the Solicitor General).
-  Lazarus, supra note 26, at 1497.