We're all familiar with the normal way in which NFL statistics are expressed. A statistic is given, and then you are told where that statistic ranks against all other teams that season. This is the way we've always done it, and it's likely to be the way we will always do it.
But just because we've always done it that way doesn't mean that it's the best way.
Here's the problem: statistical rankings don't tell you anything about the quantitative difference between teams with adjacent (or similar) rankings. To illustrate, let's say that in a particular season, three teams tie for first place in rushing yards per attempt at 4.9. Those three teams are therefore ranked #1. The next-best team has 4.8 (ranked #4, because three teams tied at #1) and the one after that (ranked #5) has 4.0. So the difference between the first ranked team and the fifth ranked team is almost a yard. So far, so good, right? But suppose, though, that this third-best team has 4.7 yards - just two tenths of a yard behind the first-ranked teams - and is still ranked fifth.
Does that make any sense?
Of course... while such a thing could easily happen, this is just a made-up example to illustrate a point. So let's examine a more likely one:
Let's say that the team ranked first has 5.3 yards. The number two team has 4.9. Number three has 4.8. Three teams are tied at fourth with 4.6, three more are tied at seventh with 4.5, two are tied at tenth with 4.4, and four more are tied at twelfth with 4.3.
For the purposes of our illustration, let's say that the following season, the number of yards gained by the top 15 teams are distributed the same way - the top team has 5.3, the second has 4.9, and so on.
But it's a new season, and one of the teams that was tied for twelfth the previous season is now tied for fourth. Their fans exclaim, "Wow! We went from twelfth to fourth in rushing yards!" Men cheer. Women weep. Kids take the day off from school for the parade!
Their team improved three tenths of a yard. Meanwhile, one of the teams ranked fourth has moved into second place - advancing just two positions in the ranking - by increasing the same three tenths of a yard.
Does that make any sense?
(By the way... these numbers aren't just a more likely example. These are the actual rushing yards per attempt for the top 15 teams in the NFL in 2016. I just made up what happens in 2017 because... well... the 2017 season hasn't happened yet! But I can guarantee you that after next season, some fanbase is going to be celebrating an empty victory based on a big jump in a stat ranking. And another is going to say, "Meh," because their team only moved up a spot or two)
Golly... if only there was a way we could compare statistics with one another - one that allowed us to not only easily see which ones are better than the others, but also reveals quantitative differences between them!
Guess what? There is a way. Statisticians do this by calculating standard deviations from average.
Now... don't be skeered. This is simply a mathematical method to determine whether one value is significantly different than another - that is, whether the difference between them is statistically significant. Using standard deviations from average, we can tell whether a particular value falls within a specific range compared to the average. To put it another way, values within one standard deviation of average are in one range, those between one and two standard deviations are in another, and those between two and three are in yet another. Therefore, a figure representing standard deviations from average is a number that usually falls between three and minus three - for example, -1.07, 2.51, -2.15, 1.98 and so on.
I'm reading your mind. "Geez, Dixon... you aren't REALLY suggesting we substitute all this math gobbledegook I don't understand for the team rankings I have used all my life, are you? I'm going to have to understand what all this standard dev-le-actuary crap means?"
Nope. You're not. Because you already understand it.
You learned it in high school, when your teacher announced that next Tuesday's test would be graded "on the curve." You might not have understood what they meant, but during lunch - in exchange for your corn dog - the geeky math whiz with the pocket protector explained the practical effect: a few in your class would get As (or Fs) a somewhat larger number would get Bs (or Ds) and about two thirds of the class would get Cs.
But here's what was actually happening: the teacher was simply awarding everybody a letter grade based on how many standard deviations their score was from average - like this:
You've probably seen one of these bell curves before. I had, too, but until I began researching this idea, I had never connected the phrase "standard deviations from average" with it. Your teacher didn't just decide at random where the cutoffs between grades were supposed to be. Those were determined by standard deviations from average.
If you're s sharp-eyed reader, you will notice that no teacher ever gave F grades with a plus or minus - or, for that matter, a grade of FF or AA. (We'll get to that in a minute) All you need to know right now is that we don't have to utter the phrase "standard deviations from average" ever again. We'll just take a cue from your high school teacher, and express numeric figures for standard deviation from average as letter grades that everybody understands.
Feel better now?
So what happens when we apply all this to our second example - Rush Yards/Attempt in 2016?
Team | 2016 Rush Yds/Attempt | Rank | Grade |
Buffalo Bills | 5.3 | 1 | A |
Cleveland Browns | 4.9 | 2 | B |
Dallas Cowboys | 4.8 | 3 | B |
Atlanta Falcons | 4.6 | 4 | C+ |
Tennessee Titans | 4.6 | 4 | C+ |
Chicago Bears | 4.6 | 4 | C+ |
Green Bay Packers | 4.5 | 7 | C+ |
Washington Redskins | 4.5 | 7 | C+ |
Miami Dolphins | 4.5 | 7 | C+ |
Oakland Raiders | 4.4 | 10 | C+ |
San Francisco 49ers | 4.4 | 10 | C+ |
New Orleans Saints | 4.3 | 12 | C |
Arizona Cardinals | 4.3 | 12 | C |
Pittsburgh Steelers | 4.3 | 12 | C |
New York Jets | 4.3 | 12 | C |
Kansas City Chiefs | 4.2 | 16 | C |
Jacksonville Jaguars | 4.2 | 16 | C |
Philadelphia Eagles | 4.1 | 18 | C |
Houston Texans | 4.1 | 18 | C |
Indianapolis Colts | 4.0 | 20 | C- |
Carolina Panthers | 4.0 | 20 | C- |
Baltimore Ravens | 4.0 | 20 | C- |
Cincinnati Bengals | 4.0 | 20 | C- |
New England Patriots | 3.9 | 24 | C- |
Seattle Seahawks | 3.9 | 24 | C- |
San Diego Chargers | 3.8 | 26 | C- |
Detroit Lions | 3.7 | 27 | D+ |
Tampa Bay Buccaneers | 3.6 | 28 | D+ |
Denver Broncos | 3.6 | 28 | D+ |
New York Giants | 3.5 | 30 | D |
Los Angeles Rams | 3.3 | 31 | D- |
Minnesota Vikings | 3.2 | 32 | F+ |
Be honest: if you held your hand over the right hand column - looking only at the rankings - you'd probably say that the Bills, Browns and Cowboys are elite at rushing the ball. They are, in fact, the top three teams in the league. But when we apply letter grades, we see that only the Bills are really elite - that is, head and shoulders above everybody else. The Browns and Cowboys are simply above average - and the three teams tied for fourth have average grades!
By the same token, if you looked only at rankings, you could easily conclude that the Giants, Rams and Vikings are awful. Yet across the league, only the Vikings really deserve that designation; the Giants and Rams are simply below average.
But those three teams bring up an important point: there's very little statistical significance between adjacent grades. Compared to each other, the Vikings (with an F+) are only a little worse than the Rams (with a D-) so the difference between them is measurable, but not statistically significant. The same is true for the D- grade for the Rams, which is only a little worse than the D grade for the Giants; the difference between those two teams isn't statistically significant, either.. But you could say that the Broncos, Bucs and Lions - each a full letter grade higher - are all significantly better than the Vikings.
Back in high school, Mom and Dad understood these distinctions. If you brought home a C+ in English at midterm, they might not have liked it very much, but they understood that you weren't doing badly - you were just a bit above average. If you improved that C+ to a B- by the end of the year, they were happier. Maybe you got to stay out an hour later on Friday night. If you went from C+ to a B, they were a little happier - but still not that impressed. Maybe they let you take Dad's Cougar out on Friday night, instead of Mom's Ford station wagon with the wood on the sides. But you didn't get that BIG reward (letting your girlfriend spend the night) unless you went from a C+ to at least a B+. (OK... maybe I'm exaggerating a little. Maybe they let you and your girlfriend - ahem - "study" in your room)
Your parents understood how it worked: unless your grade changed by at least a full letter grade, it was a measurable difference - but not all that significant.
If you're thinking all this through, you should now see why F- and F+ grades should be included - even though your high school teachers didn't use them. Statistically speaking, it makes no sense to have A- and A+ grades if you're not also going to have F- and F+ grades.
So what about the AA and FF grades shown on the bell curve above? That's because unlike school - where your grade had definable upper (and lower) limits - generally speaking, sports statistics don't have limits. While it will happen only rarely, it's entirely possible for a statistic to be four - maybe even five - standard deviations from average. So I've added AA and FF grades to be used whenever a statistic is at least three standard deviations from average.
***
By now, you might be a little depressed. Maybe at some time in the past, you've argued that because a team or player is in the top 10 in some statistic, they are above average. (Don't feel bad. I've done it myself!) But it looks like maybe you (and I) are going to have to rethink that. Just in the one statistic I have shown you, you can see that this isn't really true - the teams ranked from 4th to 26th all have C grades! Or maybe you've argued that because a player or team is ranked in the top 3 (or top 5) in some statistic, they are elite. Well... maybe that player or team was elite. Or... maybe they weren't.
Even so... this a different way of looking at it, and it's natural to be resistant to it. You might be tempted to take a shortcut, and translate rankings into letter grades. You might figure that the top three teams are elite, and the bottom three are awful.
Sorry... but you can't.
While this will tend to be true for all statistics, it probably isn't true for a specific statistic - especially for NFL statistics, which by their very nature have limited sample sizes. In a given stat for a given period of time, there could easily be three A grades, five F grades and one D grade - with the rest being Cs. Or there could be a couple of B grades, a D grade, but no As or Fs. You just can't depend on the grades being distributed the same way as they are in the example I have given.
You could also find this approach depressing because it might seem to suck all the fun out of it. In our example, you can see how it would be possible to improve from 26th one season to 4th in the next - but have a C grade in both seasons. Yeah... there's not much fun there. But it's intellectually honest, and statistically defensible.
***
Obviously my humble FanPost on a team sports blog isn't going to make the NFL, Pro-Football-Reference, ESPN or any other vendor of statistics see the light, substituting letter grades for rankings - although I'd be perfectly happy if that happened! But speaking for myself, I will never again write an article that depends on statistical rankings in its analysis.
And if you're one of the many people on this site who also write articles with statistical analyses, I ask you to consider using this concept in your work, too.
It's really not that hard - especially since I've made it easy for you! I have set up a spreadsheet to show how this is done. You can look at it, use it, and even download it by clicking here. As it is, you can insert any statistical data you want and calculate the letter grades. Instructions for its use are included.
And if you're handy with spreadsheets, you're more than welcome to steal everything in it, and integrate this concept into your own projects.
Thanks for reading. I hope I get an A, but recognize that a C is more likely.