An analysis of nearly 4 million pitches shows just how many mistakes umpires make

Baseball is back, and fans can anticipate another season of amazing catches, overpowering pitching, tape-measure home runs – and, yes, controversial calls that lead to blow-ups between umpires and players^[1].

Home plate umpires are at the heart of baseball; every single pitch can require a judgment call. Yet ask any fan or player, and they’ll tell you that many of these calls are incorrect – errors that can affect strategy, statistics and even game outcomes.

Just how many mistakes are made?

Comprehensive umpire performance statistics are not readily known, tracked or made available. Major League Baseball doesn’t seem interested in sharing the historical data^[2].

Could it be because the numbers aren’t flattering?

Luckily, every MLB pitch is tracked and made available – numbers then have to be accessed, downloaded, sorted and evaluated. This takes time and computing power. In a new study^[3] with support from a team of Boston University graduate students, we closely analyzed how many balls get called strikes and vice versa. The accuracy of all home plate umpires was ranked and age and experience taken into account.

While the human element of the game certainly adds color, our results show that it comes at a high cost: far too many mistakes.

Mining the data

All 30 Major League Baseball stadiums are outfitted with triangulated tracking cameras^[4] that follow baseballs from the pitcher’s hand until it crosses home plate. Ball location can be tracked up to 50 times during each pitch, and accuracy is said to have a margin of error of 1 inch. This information is used to evaluate players, but MLB doesn’t share the results in a way that allows fans to easily evaluate the performance of umpires.

We analyzed nearly 4 million pitches over the course of the last 11 regular seasons. This data, which had been collected by MLB-owned Statcast^[5] and Pitch f/x^[6], was sorted, formatted and superimposed on a standard strike zone map.

Using this available technology, we measured ball and strike calls for accuracy. We then ranked the error rates for each active umpire, creating a “Bad Call Ratio.” The higher the ratio, the worse the umpire.

The findings were troubling.

Botched calls and high error rates are rampant. MLB home plate umpires make incorrect calls at least 20% of the time – one in every five calls. In the 2018 season, MLB umpires made 34,246 incorrect ball and strike calls for an average of 14 per game, or 1.6 per inning. Last season, 55 games – 2.2% of the total played – ended with an incorrect call.

When batters had two strikes, the error rate for all umpires increased – incorrect calls happen 29% of the time, almost double the error rate when the batter had one or no strikes.

We also found that the highest error rates did not come from younger, less experienced umpires; they came from the older, veteran umpires. The average MLB umpire^[8] is 46 years old, with 13 years of experience. But the top performers between 2008 and 2018 had an average age of 33 years old and had less than three years of experience at the big league level. Like professional baseball players, professional umpires seem to peak at a certain age.

Despite years of data-driven evidence, MLB has notoriously resisted retiring poorly performing umpires and hiring better-performing ones. The league remains top heavy with aging umpires, making it difficult for fresh new talent to make impact.

Umpires can still play a role

For all of the ways MLB has incorporated technology into the game – the radar gun, instant replay, pitch graphics, Doppler radar^[9] – the league has resisted deploying this technology to assist with calling balls and strikes.

Umpires continue to call balls and strikes like they did a century ago when Babe Ruth played.

I’m not proposing that baseball bring in robots and fire the umpires; baseball has too many one-off situations and complexities to assume a bot could replace an umpire. But MLB does have a unique opportunity to use existing technology and strengthen human-software collaboration so umpires can do a better job.

Umpires could easily be fitted with ear pieces connecting them to a control center that conveys real-time ball and strike information. These tech-assisted umpires could then make calls correctly, quickly and effortlessly. Time-honored and much beloved behind-the-plate signs, signals and sounds would still exist. And umpires could remain the final arbiter, having override ability under certain circumstances, such as if a ball hits the ground before crossing the plate or if a system outage occurs.

Strong recruiting, hiring and retention of superior performing umpires coupled with tech aids would reduce error rates and also help dampen biased pitch calling. Strike zone subjectivity would be minimized, allowing batters and pitchers to focus more on their craft and less on guessing a specific umpire’s strike zone quirks. It would also reduce conflict between teams and umpires. And imagine how much the player and fan experience would improve if more than 34,000 annual incorrect calls vanished.

References

1. ^{^} to blow-ups between umpires and players (youtu.be)
2. ^{^} doesn’t seem interested in sharing the historical data (www.nydailynews.com)
3. ^{^} In a new study (www.bu.edu)
4. ^{^} are outfitted with triangulated tracking cameras (m.mlb.com)
5. ^{^} Statcast (m.mlb.com)
6. ^{^} Pitch f/x (library.fangraphs.com)
7. ^{^} Pitch F/X (www.boston.com)
8. ^{^} The average MLB umpire (mlb.mlb.com)
9. ^{^} Doppler radar (spectrum.ieee.org)
10. ^{^} AP Photo/Ralph Freso (www.apimages.com)