“On a historical basis, a decade from now, we’ll be looking back saying, ‘That was the highest route efficiency that’s ever been captured in baseball.’”
That’s what Joe Inzerillo—the executive vice president and chief technology officer of MLB Advanced Media—said in a league press release announcing baseball’s revolutionary new player-tracking system, Statcast. It hasn’t quite been a decade since that quote; it hasn’t quite, in fact, been three years. But route efficiency, the metric in question, has already disappeared.
Using a combination of cameras and radar to track the ball in every position it reaches as well as every player on the field at all times, offering a theoretically perfect or at least perfectible view of every game played in every major-league season, Statcast offers nearly limitless possibilities for baseball analysis by producing an incredible amount of raw data, some of which is packaged into specific metrics designed by a team of people at MLBAM. There are statistics meant to help others make sense of the information, statistics meant to work as storytelling tools, statistics meant to create additional layers of meaning and context for this vast new knowledge base, and statistics meant to fulfill varying combinations of these functions. When Statcast was rolled out across the league and to the public in 2015, route efficiency was among the most prominent of these metrics, aiming to capture, more or less, just what its name described—how efficient a route an outfielder took to get to a ball, with 0 percent being the least efficient and 100 percent being the most.
Throughout 2015 and 2016, route efficiency was everywhere: on national broadcasts, on the league’s social media accounts, in articles on MLB’s website. And then, before the start of this past season, it quietly vanished. MLB stopped publicly mentioning it, and then removed it from the online glossary of Statcast terms. Instead, they started championing a new and improved outfield defense metric: catch probability, which uses the distance needed to get to a ball and how much time the fielder has to get there to figure out how likely it is to be caught.
For many fans—most of those inclined to care, maybe—the switch from route efficiency to catch probability was a mere blip on the radar, if it was even noted at all. For MLBAM, it was an example of the iterative process needed to find the best way to understand this new data and share it in a way that’s compelling and meaningful while still being accessible to all different sorts of fans. And for a small but fervent group of independent amateur baseball analysts, it was a serious transgression: an attempt to scrub public work without open discussion of its flaws and, more importantly, without releasing the underlying raw data used to build the metrics in the first place. It was an unequivocal step forward in the eyes of MLBAM—they’d discovered a way to improve their work, and so of course they’d taken it. But for some hardcore sabermetricians and members of their adjacent communities, progress could not and cannot be meaningful unless built on the principles that have traditionally guided public baseball research: transparency, open methodology, and the capacity for and encouragement of peer review.
For some hardcore sabermetricians and members of their adjacent communities, progress could not and cannot be meaningful unless built on the principles that have traditionally guided public baseball research.
Route efficiency is a tiny sliver of what Statcast has produced so far, and a much tinier sliver of what it can potentially produce in the future. But it’s indicative, more broadly, of one of the toughest questions that Statcast faces: How can a league-owned and -operated system entertain and serve branding needs while also producing cutting-edge research? Further, what does the answer to that question mean for the state of public baseball analytics writ large? Statcast—a closed, proprietary system that could serve as the final end of a traditionally open-access and community-based search for knowledge—has the capacity to answer sabermetrics’s most meaningful questions. If it does, it will not do so by following the sabermetric movement’s traditional path.
To understand why analysts care so deeply about Statcast, it’s first important to understand just how it works and what it can do. The system has two parts: cameras, used to track primarily the players, and radar, used to track the ball. The high-resolution cameras capture stereoscopic video and come from the broadcast graphics and data visualization company ChyronHego; the radar follows the ball by tracking the seams at a rate of 20,000 frames per second and comes from the Danish company TrackMan. This combination yields an incredible wealth of information about a single play, to say nothing of an entire game, all of it described with a unique technical vocabulary. You have precise information on where the pitcher released the ball (extension), how hard he threw (velocity), what sort of spin he had (spin rate), and how fast it appeared to the batter (perceived velocity). You have how hard the ball was hit (exit velocity), how it came off the bat (launch angle), and exactly where it went (batted ball direction). You have where the fielders were positioned when the pitch was thrown and how they moved as it was hit; you have how hard an outfielder made a throw (arm strength) and how fast the runner was going (sprint speed).
Among all this you have, essentially, a way to quantify just about every aspect of the game, no matter how minute. The existence of the Statcast system inherently revolutionizes baseball statistics and analysis and, in turn, baseball itself. The sport’s advanced metrics have traditionally been about finding ways to convert player performance into specific quantitative measures of value: runs, wins. Statcast offers a new framework, allowing you, and the analyst, and the ballclub, the opportunity to ask and answer fundamentally different questions. Existing metrics typically examine what the final outcome was, what the final outcome should have been, who was chiefly responsible, and what all of this was worth, as denominated in approximations of values that have approximate values relative to wins. These questions, and the ideas behind them, are valid and important in their own ways, but Statcast is able not just to build on this structure, but to shift it to another dimension entirely. Statcast has the power to answer how a play happened and why a play happened and what it meant, specifically, not as an abstraction but as a thing in its own right.
Every major league team does its own private analysis of this data in-house, and the specific language of Statcast is increasingly replacing the more general and traditional baseball vocabulary in everything from front-office announcements to player reactions. “Elevate the ball” has been tossed around just about as long as baseball has existed, but when Ryan Zimmerman hit a game-winning home run against the Chicago Cubs in Game 2 of the National League Division Series this year, teammate Bryce Harper mentioned the “great launch angle” in his post-game comments.
“It would’ve been unfathomable, five years ago, to say that basically one of the top five players in the game would be using a term like that earnestly to describe a high-profile play,” said Matt Meyers, senior director of content for MLB’s website and one of the hosts of the official Statcast podcast.
What teams are doing on their own with the data to select players, evaluate them and coach them is—like any in-house analytical work—kept private, part of a fight for competitive advantage. But how Statcast is used in public falls to Meyers and his coworkers. (That team that will soon include Ben Jedlovec, former president of top-tier stats company Baseball Info Solutions, who recently announced that he’ll be coming onboard in January.) They’re charged not just with figuring out how to use the data for compelling research of baseball’s biggest questions, but also with figuring out how that research can be relayed—or translated—to fans in ways that are interesting and entertaining but don’t sacrifice meaning. The framework that they’ve built heavily influences how Statcast is used on broadcasts, on league and team social-media accounts, and on ballparks’ video screens during games. It’s an inherently difficult job, and especially so when one considers the breadth of fans they’re talking to: those who hold fast to old-school figures like runs batted in, those who read sabermetric websites daily, and everyone in between. For some fans, their work might come off as an unwanted math lecture in the middle of a game; for others, it might seem like an endeavor that can never be intellectually or technically rigorous enough.
“I try to think of it in a way of, How can I write this in a way that my dad might like it?” said Mike Petriello, an MLBAM analyst hired specifically to work with Statcast. “He’s a smart guy, a baseball fan, but he’s not super into all the crazy numbers. That’s always the interesting part for me—how do I balance both of those fanbases?”
Some of the features that have gotten the best feedback from fans are raw figures that tap into a basic baseball framework that most fans should already have. Exit velocity, for instance, is conveyed in a way that everyone gets—miles per hour—and on a scale that’s pretty easy to understand. A guy hitting the ball 100 mph is hitting the ball hard, and people can see it with their own eyes. The same goes for something like arm strength. In this sense, Statcast offers something much more closely tied to actual baseball activity than do most of the sport’s other numbers. These figures don’t require getting tangled in the conceptual structures that provide a foundation for many other metrics, those seen when estimating a theoretical number of runs allowed above or below the average fielder, in UZR (ultimate zone rating), or distilling the many varied aspects of play into a single figure to measure total performance comparative to a replacement player, in WAR (wins above replacement). Statcast is quantifying features that are much more straightforward and tangible and, quite simply, much more centered on baseball as it’s played on the field. These figures don’t reflect how a player registers against all other players ever as measured in a theoretical vacuum; they tell you how hard a player threw, how fast he ran, how the ball came off his bat.
“It’s really just getting back to the things you see on the field,” Petriello said. “You can’t see a weighted run created plus. You can’t say, I saw that. But you can say, I saw Jake Marisnick or whoever throw the hardest ball from the outfield all season long, or, I saw the fastest inside-the-park home run that’s ever been tracked. So I think in that sense, you don’t have to overcomplicate it. You can say the fastest, the best. You’re just putting numbers to it.”
“It’s really just getting back to the things you see on the field.”
Much of what Statcast offers, though, is far more complex than these comparatively straightforward figures. Tackling a multi-faceted issue like, say, how to evaluate outfield defense has involved packaging data into new metrics that combine meaningful pieces of the relevant information into one figure. This is where a concept like route efficiency or catch probability comes in, and it’s where Statcast has to answer some of its biggest questions.
First, there’s the question of how they decide which ideas to tackle, and in which order they do so.
“We all have things we really want to do. It kind of comes down to what’s important for those above us,” Petriello said, referring to higher-ups at MLBAM, such as vice-president of analytics Cory Schwartz. “And then—not how simple, but how doable is it for us to do those things? Is this a two-week effort, or is this an eight-month effort? … It’s nice to be able to stagger it sometimes.”
(Some projects currently in the works include a sacrifice fly model to determine whether a team should have sent the runner and a metric to analyze a catcher’s responsibility for stolen bases.)
Then there’s the issue of how easily these concepts can be presented to and understood by fans. It’s one thing to consider how the metrics will be used online, which offers the benefit of unlimited space for explanation, and the convenience of being able to link to the Statcast glossary maintained by MLBAM. But they also must weigh how the metric will come across in a context that’s far more constrained. Like, say, a 45-second replay segment on a national broadcast.
“That’s sort of the toughest nut to crack—finding ways to get it on broadcasts,” Meyers said. “Because it’s got to be quick and it’s got to be easy to contextualize and it’s got to be something that the commentators will take to and buy into. So you need all these things, and a big part of the challenge is creating these metrics and tools that we know can be used in real time.”
Finally, there’s the question of how exactly to go about formulating a metric—a process that changed significantly with last year’s hiring of Tom Tango as the project’s senior database architect.
Tango is among the most prominent of the first wave of online baseball analysts, and his individual rise loosely reflects that of the larger movement. He started out the same way that most people did in online sabermetrics’ foundational age, about two decades ago—connecting with other statistically-inclined fans on message boards, analyzing Retrosheet’s public collection of box scores, and forming a community through sharing and discussing research. One of his most significant breakthroughs came from building off a discovery by fellow amateur analyst Voros McCracken, whom Tango met on the now-defunct baseballboards.com. He took McCracken’s brainchild of defense-independent pitching statistics a step further by developing fielding-independent pitching, FIP, a now-popular metric that aims to improve on ERA by separating a pitcher’s individual performance from the work of his defense.
That early public analytics community grew, with people sharing their work so that others could debate it or corroborate it or rip it apart as they saw fit. And as that body of internet research became more robust and broke more ground, its ideas were noticed by teams. The message boards’ brightest minds increasingly got the chance to enter front offices, and so the outsiders became insiders.
Tango realized that teams were paying attention to his research when Moneyball author Michael Lewis called him up to say that the A’s front office was reading his work, and he’s since done consulting for several MLB teams. His current job, however, is his first full-time position in baseball. By bringing him onboard, MLBAM invested in someone who had once been a leading figure not just in baseball research, but specifically in open baseball research: someone who’d kept a public analytics blog running through his years of private consulting, who’d explained relatively complicated statistics to ordinary readers as an author of popular saber volume The Book, and who’d consistently and strongly advocated for the open-source environment that created the early years of online sabermetrics.
Tango’s job is now to use MLBAM’s private and proprietary data to create public statistics, and both he and his colleagues say that he’s changed the model for generating metrics quite a bit.
“I think that really, in the early years of Statcast, it was, Let’s calculate all these things and let’s publish them and then try to figure out what it means after,” Tango said of the process in the two seasons before he was brought onboard. “And I think that’s where with route efficiency, you got kind of stuck—where it seemed natural to do it the way they did it, but then once you see the result on a large scale, you say, Well, okay, maybe not. So then you have to take a step back and say, Now we’ve really got to figure out how to do it.”
One problem with route efficiency was that almost every single route fell within a narrow range of 90 to 100 percent, making it difficult to contextualize and show meaningful differences. Another issue was hardwired into the metric’s definition: A perfectly efficient route isn’t always the best one. If an outfielder circled behind a sacrifice fly ball to make a better throw, for example, he’d be penalized for his lack of efficiency, even though the seeming inefficiency was necessary to make the play in the first place. The updated version, catch probability, addresses this by asking a different question, one without a subjective ideal like efficiency embedded in its foundation. It’s not asking how economical a fielder’s path to the ball was, in an environment where economy of routes can arguably take significantly different forms; it’s asking how likely the ball was to be caught, using the specific data that exists for similar catch opportunities as a comparison point.
MLBAM says that they want to revisit route efficiency, in some form, in the future. This research is a gradual and iterative process, after all, and they’re moving one step at a time. But for some in baseball’s small-but-passionate community of independent public researchers, the league’s rush to promote the metric before they realized its flaws shows a serious reason for concern.
“I think they thought this would be easier than it is, and it just isn’t,” said Harry Pavlidis, director of technology for sabermetric website Baseball Prospectus and founder of the pitch-tracking company PitchInfo. “I don’t think they had the right decision structures in place in terms of deciding what was a minimum marketable product.”
(Disclosure: I wrote for Baseball Prospectus during the 2016 season and currently contribute to their weekly shortform series.)
Traditionally, most of the major developments in public baseball analytics have come from individual researchers unaffiliated with the league. Decades ago, this was because those individual independent researchers were usually the ones collecting the necessary data in the first place. MLB has long since surpassed these hobbyists in terms of collecting game data, of course. But they’ve typically made that data public. Even in baseball’s last big technological leap forward—the pitch-tracking system Pitchf/x, installed in big-league ballparks in 2008—all resulting information was released for outside analysts to work with. But that hasn’t been the case with Statcast, which has left many outside researchers somewhat frustrated and suspicious. Certain portions of the data have been released directly, such as the exit velocity and launch angle of batted balls, and more can be gathered from metrics like catch probability. But the complete raw data is still a black box, which makes it hard to scrutinize MLBAM’s metrics, and can make it confusing when those numbers are updated or even scrapped altogether. The most common complaint about the fate of route efficiency isn’t that the league was willing to experiment and play around with different potential metrics. It’s that the continued experimentation was happening behind closed doors, while early results were being publicized as high-quality tools for the public.
“There should be a higher standard,” said Rob Arthur, an independent researcher who has served as an MLB front-office consultant in the past and currently publishes his analysis at FiveThirtyEight. “I think that’s one of the ways that they’ve kind of erred at times. I don’t see any problem with playing in the sandbox, but if you’re going to play in the sandbox, you have to get all the way in there. You have to provide what’s going on and explain what route efficiency means and what it comes from and show us the raw stuff that goes into it.”
Analysts like Pavlidis and Arthur are frustrated that the data is closed not only because it makes it difficult to judge the conceptual and technical rigor of metrics, but also because it makes it difficult to judge the accuracy of the data. Statcast’s cameras and radar are advanced, but they aren’t perfect. Last August, for example, Arthur published research comparing Statcast’s public game information with statistics recorded by human stringers to show that the radar completely missed 10 to 15 percent of batted balls (mostly those with unusual trajectories, such as very high pop-ups or very low grounders). Statcast acknowledges this, and the system estimates information for missing balls by combining observations from human stringers at the park with the numbers they have on average hit trajectories. Still, the fact that Statcast is more likely to miss certain types of batted balls means that certain types of hitters are more likely to have an incomplete profile, which can create a biased data set. This led independent analyst Jeff Zimmerman to try to find and incorporate the missing data from 2015 and 2016 into his own exit velocity and launch angle leaderboards, published at sabermetric site FanGraphs last December.
That’s just batted balls. This year brought an entirely separate controversy over pitch-tracking information. Up until 2017, MLBAM’s pitch-specific data came from Pitchf/x, the camera system installed in all major-league parks for that purpose nearly a decade ago. But beginning this season, they decided to switch over from Pitchf/x’s cameras to Statcast’s radar. (The radar was already being used to track the ball in play, but any specific information about the pitch itself—such as its velocity—had been coming from the camera system of Pitchf/x.) Pitchf/x and Statcast don’t correlate precisely, though. The former measures velocity from a set point 50 to 55 feet back from home plate, while the latter measures right out of the pitcher’s hand. This means that Statcast readings will nearly always be faster, and switching from Pitchf/x readings led to some significant changes in basic pitching data. To someone who didn’t know that this change had taken place (which was just about everyone who wasn’t directly affiliated with MLB), it looked like almost every pitcher in the league had experienced a velocity bump of as much as a few miles per hour. On April 3, FanGraphs writer and analyst Jeff Sullivan published a piece noting as much; the next day, FanGraphs editor-in-chief Dave Cameron got clarification from MLBAM that the pitch-tracking system had, in fact, changed.
For many independent researchers, the lack of initial communication on the switch was disappointing in its own right, but the fact that the switch created new problems with the data was even more so. The radar system didn’t seem to be properly calibrated for pitches in every ballpark, causing measurement issues that had never really been a problem with Pitchf/x. A few weeks into the season, Arthur published work showing that errors in both horizontal and vertical pitch movement were higher under the new system than they had been at any point in the recent history of Pitchf/x. These errors got smaller over the course of the season, Tango said, and there’s now a disclaimer at the top of MLBAM’s BaseballSavant.com Statcast data search page noting that pitch velocities from 2008-16 are from Pitchf/x cameras and those from 2017 on are from Statcast radar. That clarification is informative and necessary, but the situation can still be maddening for anyone attempting multi-year analysis, to say nothing of an average fan quickly checking to see if his favorite pitcher threw any harder this season than the one before. In a league where a velo increase of even a single mile per hour can be meaningful, comparing measurements taken by different systems can feel essentially useless. This set-up makes existing research harder to build on, and it gives future research a smaller sample size from which to draw.
Statcast’s systems have been getting steadily better over time, a finding that Arthur noted in his batted-ball research and which the league emphasizes. Tango says that MLBAM has weekly talks with ChyronHego and TrackMan about the technology, as well as more frequent casual conversations about anything that seems potentially off. Some things are easy to notice and start addressing—like a problem earlier this season measuring velocity in Atlanta’s new ballpark—and others are more complex long-term issues, such as figuring out how to get the cameras to stop losing players in the shadows of the outfield.
No one expected the system to roll out with perfectly complete and accurate data right from the beginning. That’s simply not the nature of multi-sensor system tracking analysis. But the fact that most of the data have been kept private has made it difficult for independent analysts to tell exactly where and how the system is missing information. It’s a stark contrast from the introduction of a system like Pitchf/x, where public analysts were able to dig into the data and offer suggestions on areas that might need improvement—along with, of course, ideas about the best ways to use the data and the richest insights that could be gleaned from it.
“They would be better off now [if the Statcast data were open],” Pavlidis said. “Three years into Pitchf/x, we had done a lot to fix the data, and it was always encouraged.”
The company behind the Pitchf/x system (SportVision, which has since been acquired by athletic technology group SMT) actively engaged with independent researchers who were working with the data, inviting Pavlidis and others out to conferences to present their findings. There haven’t been any similar moves with Statcast—though, with MLBAM’s decision to keep the complete data set private, it’s hard to imagine that researchers would have nearly as much work to present as they did in a situation where they did have that access, like Pitchf/x.
“[MLBAM] might be able to reach better metrics more quickly if the data were all publicly available. There would be an army of amateurs out there—very talented amateurs, I might say—that would work on developing their own metrics,” said Dr. Alan Nathan, professor emeritus of physics at the University of Illinois and a baseball analyst who has published research with Statcast data and done extensive work on the science of the sport. “That’s how Pitchf/x got developed. The data were completely public and MLBAM, I think, benefitted a lot from people who were moonlighting and doing this analysis in their spare time. They benefitted tremendously from that collective wisdom that sort of developed.”
There are clear parallels that can be drawn between Pitchf/x and Statcast, but it’s unfair to make a one-to-one comparison. The former is a fairly large and detailed dataset; the latter is incredibly, dramatically more so. Overall, including the raw video, Statcast produces several terabytes of uncompressed data per each individual game. (That’s more raw data than the Library of Congress adds to its web archive each month in just one game.) The final stored statistical data is far more manageable, at 250 megabytes per game without video. But when you’re talking about a full season, that still creates a set that would be far more difficult for amateur researchers to work with than was the case with Pitchf/x.
In the context of the total amount of existing Statcast data, what’s been publicly released is only a very small slice. But compared to what baseball fans had to work with just a few years ago, it’s a significant upgrade.
“I certainly understand why everybody wants everything,” Petriello said of independent analysts’ desire for open data. “But I would hope that people find it cool that—just three years ago, thinking that you would have the exit velocity and the launch angle for every single batted ball, knowing the speed of every single player!—there’s a lot of stuff that’s out there.”
It is a lot. Just a few clicks on Baseball Savant, the website that houses the data, can give you everything from sprint-speed leaderboards to the quality of contact for any batted ball. But it won’t give you everything, and when it comes to the future of a public analytics community that’s been built on reviewing and critiquing the analysis of others, that is worrisome.
“In the long term, I think it risks choking off the public analytics community, because we won’t be able to have the same quality of data that the people in the league and the analysts working for teams have,” Arthur said. “We won’t be able to scrutinize their decisions or even understand what they’re doing. I do worry that, in the long run, we’re not going to make as much progress because this data is absent or is being tightly controlled to the point where we’re not able to look into it.”
“I do worry that, in the long run, we’re not going to make as much progress because this data is absent or is being tightly controlled.”
MLBAM counters this idea by saying that there are still plenty of opportunities for the public to conduct meaningful research with the presently available data alone. The information on launch angle and exit velocity, for instance, might be used to analyze hitting in any number of ways that people haven’t even begun to dream of yet.
“Maybe it’s too much data that they don’t know where to start,” Tango said about the current state of research with public Statcast information. “There’s so much available already, so little’s being done with it, and we keep giving more and more. I don’t know that it’s important that we have to dump the whole set right now and overload even more.”
For independent analysts, this is a question of principle more so than it is one of simple practicality: It’s not so much about what has been done already, but rather about what could be done in an environment where hundreds of different minds are looking at the data and bringing fresh perspectives to the table. It isn’t that public researchers believe working with the data might allow them to find new answers to MLBAM’s Statcast questions, though that’s certainly true. It’s that they very well might think to approach the data with new questions altogether.
With a data set as large and complex as Statcast, however, the principles of an open-source ideal do have some practical constraints. The sheer size of the thing means that meaningful analysis would be far more complicated and demand more of amateur researchers than has been true of other baseball data sources. (This is to say nothing of the computing power that might be needed to access and manipulate a full data set: depending on the form that such information took, just downloading it to a personal computer could take hours.) That doesn’t mean that there aren’t any independent analysts with the experience, skill set, and equipment to tackle Statcast—of course there are—but it does mean that there are fewer of them than was the case when it came to, say, the innovative public work done with Pitchf/x.
While Pitchf/x was made more public in the sense that its entire data set was released, it was never a branded public entity in the way that Statcast is. Statcast has its own social media presence, a designated podcast, prominent placement on broadcasts and its own corporate sponsor—Amazon Web Services, which provides the system’s data storage. Statcast is more ambitious than anything MLB has ever done before in terms of the sheer amount of information collected, but it’s just as ambitious in terms of who it’s trying to reach: anyone and everyone who likes baseball.
The vast, vast majority of people who like baseball are not people doing independent analysis with strong concerns about data precision. The concerns of these researchers are deeply valid—wanting to be able to trust that data is complete and accurate, and that metrics are well-formulated, is wanting the system to work at the highest possible level for everyone who engages with it. But MLBAM isn’t working with the singular goal of being a data provider, and Statcast isn’t working with the singular goal of being an engine of hardcore research and analysis.
“I’ve realized that I’m not the audience,” Pavlidis said when asked about how his view of Statcast has changed over the three years that it’s been fully operational. “I tell people I work with: this is not for you.”
There is, in one sense, something disheartening about hearing a notable figure in baseball’s public analytics scene say that he realizes the game’s most remarkable analytic tool is not for him. But in another sense, one that is far more pragmatic: It’s not, and of course it’s not. A tool that’s working to engage the average baseball fan in a brief broadcast segment will naturally operate far differently than a presentation at the Society of American Baseball Research’s annual analytics conference. That’s not to dismiss the legitimate critiques of the data collection itself or to say that there wouldn’t be meaningful benefits to making the data public. But the way that Statcast is used in, say, a quick highlight on a Jumbotron is naturally not going to be incredibly appealing to an experienced researcher whose mind automatically jumps to questions about the margin of error.
Take the scale that MLBAM has developed for catch probability: rating catches as one, two, three, four, or five stars depending on how likely how they are to be made.
“We’re all like, We want a continuous variable that displays probability,” Pavlidis said of how his fellow analysts react to seeing an action so complex and affected by so many factors as a catch reduced to a simple label like four stars. “But that doesn’t matter—if they want to present it as four stars, that’s great. As long as what’s under the hood’s good, that’s awesome.”
For MLBAM, the relative simplicity of a label like four stars is actually the ideal, rather than a tolerable side-effect. The people working on Statcast see this scale as a notable success, one that takes a concept that might feel abstract and makes it easy to understand. Saying that a given play had a 44 percent catch probability doesn’t mean anything in its own right; anyone can tell that a four-star catch is pretty good. There’s a level of accessibility there that’s unmatched by other, clunkier defensive metrics.
“We think in terms of UZR, where it’s a +0.82 play, and we understand those who follow or are deep in it know what that means,” Tango said, citing one of the most prominent defensive metrics developed before Statcast. “But it’s hard to convey that kind of number—it gets lost in all the decimals. Then we can say, Byron Buxton has 29 four- and five-star plays, Ender Inciarte has 23 four- and five-star plays. Now this becomes a number that can actually relate to a physical, tangible number. And we’ll remember that, too, the way we’ll remember the count of home runs and the count of wins.”
Like anything that Statcast develops, the labels four stars and five stars aren’t meant to end the conversation. They’re simply meant to start it, or to add another layer, or to provide some statistical context: to work as a meaningful component of a baseball discussion, not to be the discussion itself. “We’re trying to find ways to make sure we highlight it when it really tells a story—to use it as a tool, not a blunt instrument,” Meyers said. The goal is to make Statcast and its metrics a natural, meaningful part of the landscape of baseball fandom for anyone who wants it.
Catch probability itself is still a work in progress. When it was rolled out at the start of the season, it didn’t yet account for fielding direction (in regular-baseball-fan-speak: whether or not the outfielder had to go back on the ball, heading away from home plate). The metric was updated in May to include that feature. And it still doesn’t fully account for whether or not the fielder has to play the ball off the wall, which is something that MLBAM hopes to be able to incorporate this winter.
“If I wanted to make it perfect, I’d have like 15 different components,” Tango said about catch probability. “It would probably take me nine months just to do this, to get it right. Or we could take the big leap forward, show that off, and then we do improvements as time allows and as we prioritize every other thing that we want to do as well.”
That’s notably different from the operating procedure that’s fairly standard in the public analytics community: typically, if a researcher knows something he’s working on is incomplete, he’ll want to, well, complete it before its release (or at least slap a “beta” label on it). But as Tango notes, the incentives and big-picture goals of a complex enterprise like Statcast are dramatically different from those of a guy moonlighting on Baseball Prospectus. And while independent researchers often measure the system by the same standards as any other project in public baseball analysis—or by even higher ones, given the involvement and investment of the league—it’s simply not any other project in public baseball analysis. It needs to speak to a far larger and more varied audience, and it needs to be available in far more forms, and it needs to work on a timetable that incorporates multiple partners. Statcast isn’t aiming for just public baseball analysis, but public baseball entertainment, too.
The forefront of baseball analytics has traditionally been something that fans chose to access. In the ’60s, they chose to read Earnshaw Cook; in the ’80s, they chose to buy Bill James’s abstracts; in the ’90s, they chose to hang out in the rec.sport.baseball group on Usenet; today, they choose to access the stats on FanGraphs or Baseball Prospectus. These individuals and communities on the margins of the game have grown and advanced wildly over the decades, and their insights have had profound implications for how front offices think about the game and what teams research privately. But the mainstream pillars of public baseball statistics—the numbers relayed in box scores, on broadcasts, on the backs of baseball cards—have largely stayed the same. Unless a fan chose to go seek out something else, they got batting average and pitcher wins and very little past that.
Statcast changes that model. Statcast is not something that a fan has to seek out—it’s just there. It’s there in any national broadcast, it’s there on teams’ social-media accounts, it’s there in everyday interviews with players. The scope is remarkable, and also somewhat complicated. It gives Statcast a platform immeasurably larger than that of any other analytic endeavor in the game’s history, and that requires the information be accessible in ways that previous iterations of baseball analytics never had to worry about.
“It sounds maybe lame to say, but it’s for everyone,” Meyers said. “There’s a huge range of who can be served by it—teams looking for a competitive edge, that’s obviously on the extreme end of it, but it’s for the casual fan, too. That’s sort of what we’re really trying to do, to create these tools that can engage casual fans and find ways to help them enjoy the game more, even if they don’t necessarily think they want it.”
Statcast is for everyone, and that means everyone can and might have a problem with it. But that also means that it’s something unprecedented in not just baseball analytics, but baseball itself.
“I have all these issues with certain things, and I would do this and that differently,” Pavlidis said. “But also, my God! Look at this. This is amazing! There’s all this incredible data they’re giving us for nothing—nothing! It’s free … This is a gift.”