Arrowhead Pride True Power Rankings: Full methodology

Over the years, Arrowhead Pride’s weekly rundown of power rankings have become must-reading for me during the season. I know perfectly well that power rankings are meaningless, since each one represents nothing more than the opinion of some NFL pundit.

But I can’t help myself. I read them anyway.

Going into the 2017 season, I started wondering: Is there a way to do it better? Is there a way to make power rankings more meaningful? And wouldn’t it be nice if you could look at them side by side, and more easily compare rankings from different sources?

So I spent the 2017 season collecting data and experimenting with it, revising my methods and calculations as I went along.

The result is Arrowhead Pride True Power Rankings — a more complete look at the state of the NFL each week. It comes to you in two parts, published on Tuesdays and Wednesdays.

---

Part 1: Pundit Power Rankings

Published every Tuesday during the regular season

On Tuesdays, we give you Pundit Power Rankings (that is, power rankings based on the opinion of a writer of group of writers) from a variety of sources, including each pundit’s comments about the Chiefs. The rankings are presented side-by-side, so you can more easily compare how different writers view each NFL team.

Then we add an extra feature: we aggregate the rankings into a single ranking based on the power rankings from ESPN, Sports Illustrated’s MMQB, NFL.com, SBNation.com, the Sporting News, Yahoo.com, CBSSports.com and USA Today.

Why these particular eight?

All are well-respected national sources that we know will publish power rankings each week. The rankings published by two of them — ESPN and MMQB — represent the opinion of all their writers, rather than just one. Since we could consider their group opinion as more valid, our calculations will weight those two more heavily than the other six; each counts two and a half times as much as any of the the single-opinion rankings.

Here’s what that looked like after Week 4 of 2018:

Note that the Week 5 power rankings are the ones published between the time the Week 4 and Week 5 games are played. For brevity’s sake, we’re just showing you the first 16 teams here; weekly Pundit Power Rankings include all 32 teams.

The leftmost column shows the aggregate ranking for each team (with the previous week’s ranking shown in parentheses), and then the individual pundit rankings are shown.

Like all the tables we show you each week, rows for the Chiefs will be in one color, rows for AFC West teams will be in another color, and rows for the remaining AFC teams will be in yet another. This immediately draws your eye not only to the information about your favorite team, but also to the teams that will most concern the Chiefs during the regular season: their division and conference opponents.

We also repeat the header row every eight rows, which will make it a little easier for you to read the table as you scroll through it — and also neatly divides the 32 teams four groups of eight.

Please note:

Since different sources publish their power rankings at widely varying times — ranging from almost immediately after the Monday night game(s) have been played to sometime around noon on Tuesday — Pundit Power Rankings start out as a live update page on Tuesday mornings. We open the live update page as soon as we can - typically between 9:00 and 10:00 am Arrowhead Time — and add new rankings to it as they become available.

This means that the aggregate rankings can (and do) change as we go along — especially since MMQB (one of the rankings based on a poll of writers) is typically one of the last to be released. Therefore, we don’t provide any analysis of the rankings until all eight have been published — and so, until the Pundit Power Rankings page shows all eight sources, the jury is still out.

Part 2: True Power Rankings

Published every Wednesday during the regular season

There are other ways to evaluate the relative strength of NFL teams — ones that do not depend on a sportswriter’s opinion, but instead on hard data.

So on Wednesdays starting after Week 2, we give you Cruncher Power Rankings generated by four well-known methods used by number-crunchers:

• Elo ratings published by FiveThirtyEight.com. This system uses points scored (for and against) as its base data, and takes the strength of the opponent into consideration. Elo (named for its inventor Arpad Elo) was originally developed to rate chess players. It is the only system we’re using that — like a pundit — considers how good a team was in the previous season, and is also the only one that cares about the results of individual games; when a team wins a game, its Elo rating always rises, and when it loses, the Elo rating always falls. Learn more by reading the full methodology or an entertaining FAQ.
• DVOA ratings published by FootballOutsiders.com. DVOA (for Defense-Adjusted Value Over Average) compares the result of every play in every game — taking into consideration opponent strength and game situation — to the league average for that set of circumstances, and then compiles this data into an overall percentage (positive or negative) that shows how one team compares to average. This is the only system we’re using that does not use points scored in a given game as its base data. Read their Methods to our madness page.
• SRS published by Pro-Football-Reference.com. Like Elo, SRS (for Simple Rating System) uses points scored (for and against) as its base data, and considers the strength of the opponent. Like DVOA, it is expressed as a positive (or negative) number that uses the league average as its baseline. Read an explanation.
• PE, or Expected Wins. PE (for Pythagorean Expectation) was originally developed for baseball by statistical guru Bill James. It is the simplest system we’re using, using points scored (for and against) as its base data, but without considering the strength of the opponent. It is expressed as a number that represents the number of games a team should expect to win in a season. Read an explanation.

None of these systems are perfect, of course. But they are — to a large extent — complementary; each has one feature that is different from all the others, and other aspects that are the same as most of the rest.

Three of them consider opponent strength, and one does not. Three don’t care which team wins or loses a given game, and one does. Three are based on points scored, and one isn’t. Three don’t consider the previous season, and one does.

See where we’re heading here? By averaging the rankings each system generates — we weight all of them equally in our calculations — we can balance out the advantages and disadvantages of each one, and come up with a Cruncher Power Ranking that is entirely data-based, which we can then compare to the opinion-based Pundit Power Ranking published the previous day.

Remember, though: for a few weeks in a new season, take what the Cruncher Power Rankings say with a grain of salt!

Why? Because there’s a strong argument to not even show you this information until about four weeks into the season. Until then, some of it will be a hot mess! Opponent strength — on which three of these systems are based — is hard to determine until about half the season is played... much less two or three weeks into a season!

For this reason, DVOA — one of the systems that relies on it — doesn’t even include opponent strength in its calculations until after Week 4, and Pro-Football-Reference doesn’t publish SRS numbers until after Week 2.

Still, we calculate our Cruncher Power Rankings starting after Week 2 anyway, because it helps give you a greater understanding of how these different rating systems actually work. The strengths and weaknesses of each one are on full display in the early weeks of the season.

Here’s a sample Cruncher Power Rankings table — again from Week 5 of 2018:

Note that in this table — or in any of our power rankings tables — there can be duplicate rankings. These aren’t mistakes. They represent ties in the underlying averages. This will sometimes happen — especially early in a season.

But we’re still only halfway to True Power Rankings. To get all the way there, each Wednesday, we simply average the Pundits Power Rankings from Tuesday with the Cruncher Power Rankings — like this:

We should note here how these rankings are averaged together.

Sometimes — especially on this table — it might seem like an average ranking has been figured incorrectly. It hasn’t been.

Here’s an example from Week 3 of 2018. The Saints were ranked ahead of the Jaguars. The Saints had rankings of fifth and eighth, which averages to 6.5. The Jaguars were ranked fourth and ninth, which also averages to 6.5. So why were the Saints ranked ahead of the Jaguars, instead of tied with the Jaguars?

Because we’re not averaging the rankings from each table. Instead, we’re averaging the numbers behind those rankings.

In the Pundit rankings for that week, the Saints’ aggregated ranking was 5.591, while their average Crunchers ranking was 10.250. That’s an average of 7.920. The Jaguars’ aggregate Pundit figure was 4.955, and their Crunchers average was 11.000. That works out to an average of 7.977. Therefore, the Saints ranked just ahead of the Jaguars in the True Power Rankings that week.

But this example perfectly illustrates the problem with any kind of ranked statistic: the ranking tells you that one value is greater (or less) than another, but not by how much.

Statisticians have a trick for this. They calculate standard deviations from average for a set of values, which is used to tell if the difference between two values in that set is statistically significant.

Standard deviations from average are expressed as a number that is almost always between -3 and 3, and is usually carried out to multiple decimal points.

But don’t worry. You don’t have to compare -1.6875 to 1.3496. Instead, we do the same thing your school teachers did: convert standard deviations from average to letter grades ranging from FF (less than -3) to AA (greater than 3) — like this:

The key thing to remember is that only a difference of a full letter grade (or more) is statistically significant.

What does that mean?

If one team has a B+ and another team has a B, the difference between them is measurable, but not significant. If one team has a C and another has a D+, the difference is a bit more measurable, but still not significant. But a team with a B+ grade is significantly better than one with a C+ grade.

We can’t do this for pundit rankings, because there is no numerical value behind each pundit’s opinion. But we can do it with the data underlying the cruncher rankings, because they are numerical values.

However, since the cruncher numbers are all based on entirely different systems, we can’t compare the raw numbers directly; that would be comparing apples to oranges. But by calculating standard deviations from average for each set of values, we effectively convert all of them to juicy red tomatoes — which we can compare directly.

Here’s how that looked in Week 5 of 2018:

We’ve shown you the whole table here so you can see how the grades tend to follow the equal distribution curve we showed you earlier. Because an NFL season consists of only 16 games, we’re always going to be dealing with small sample sizes, so you won’t always see As and Fs in the grades. But you will see a lot of Cs — just as you would expect.

These grades also reveal an essential truth about ranking statistics of any kind: numbers that we spend a lot of time comparing sometimes don’t turn out to be significantly different from each other.

While it might be true — for example — that one quarterback has a completion percentage of 60.1 and another has one of 60.4, the difference is measurable, but not necessarily significant.

Likewise... the team ranked first by the Crunchers gets bragging rights, but it might turn out they really aren’t that much better than the team ranked third — or even fifth. That’s why the Cruncher Grades are a useful reality-check each week.

---

So there you have it.

When I first published an explanation for what we were going to do with power rankings on Arrowhead Pride, I got a little bit of flak from readers who said that these rankings aren’t really “true.”

That’s a fair criticism.

But let’s be realistic: Arrowhead Pride Somewhat-Better-Than-Usual Power Rankings is hardly a catchy title, is it?

All we’re trying to do is average out the outlier pundit opinions — and cruncher scores — and come up with something that represents a clearer picture of what’s going on in the NFL as the season progresses. We believe this is a good way to do exactly that.

After all... we’re Americans, dammit! We don’t want to wait until the end of the season to know which team is the best. We want to know now!