How much better the Chiefs defense must be to win a Super Bowl

This started out as a question addressed to me for my weekly Arrowhead Pride mailbag column, but after thinking about it for a while, I thought it deserved the answer deserved an article of its very own.

What is a realistic goal for the defense in terms of improvement from 2018?

— B.S.

Here’s something many Kansas City Chiefs fans want to know: how much better does the defense have to be for the Chiefs to win a championship — and keep winning them?

On Thursday, our own Pete Sweeney asked new Chiefs defensive coordinator Steve Spagnuolo how much improvement he wanted to see in 2019 — and what “measuring stick” he would use to assess it.

“We’re a long way from that,” Spagnuolo replied. “We haven’t put a shoulder pad on. Luckily they let us put helmets on — otherwise we’d still be doing those pajama practices we were doing earlier. But it’s a long process. I’m certainly not going to compare it to anything. What I look for right now is building a foundation.

“So I’m more focused on that,” he concluded.

That’s exactly the right answer. Spagnuolo’s job isn’t to meet statistical goals. His job is building an effective defense — one that can help win championships.

But that doesn’t stop fans from wondering what it will take. Let’s take a shot at figuring it out.

---

We really have two questions to answer:

1. How good was the defense in 2018?
2. How much better does it have to be to win a championship?

As to the first question:

I’ve been beating this drum for a while: it’s a mistake to characterize the 2018 Chiefs defense as the worst in the league simply because it was 31st in total yards allowed.

The statistic is accurate, but it’s misleading because it lacks context.

As I explained when I argued it was possible Chiefs could be finished adding players to the secondary, no defense faced more passing attempts than the Chiefs. In addition, only the Cleveland Browns defense was on the field for more plays. Those two factors alone would tend to increase the total number of yards allowed.

It’s not that yards allowed don’t matter. They do matter. But volume stats paint an incomplete picture. Per-attempt statistics give you a much clearer idea of a defense’s effectiveness.

And besides... as the saying goes, yards allowed don’t go on the scoreboard. If you must use a single statistical category to define a defense, you’re much better off using points allowed.

So let’s look at points allowed (PA) alongside per-attempt statistics for yards allowed in 2018: yards per play (Y/P), net passing yards per attempt (NY/A) and rushing yards per attempt (Y/A). Teams that made the playoffs are shown in bold.

Note that I’m not showing you rankings — or even the raw numbers behind them — because they can be misleading, too. Rankings (and even the numbers themselves) tell you only that one value is greater (or less) than another. They tell you that the difference between them is measurable — but not if that difference is statistically significant.

But we can easily see this by calculating standard deviations from average, and then express those in easy-to-understand letter grades — just as I explained in an article two years ago.

So what do these grades tell us?

They tell us something you already knew: the Chiefs run defense was bad in 2018.

But they also tell us something you might not have known: that the Chiefs defense as a whole wasn’t nearly as bad as advertised — measurably less than average (a C grade is average), but not significantly less than average.

It’s simply wrong to say that the Chiefs defense “couldn’t possibly have been any worse” in 2018. Six other defenses were worse — some of them significantly so. This is because for a value to be statistically significant from another, it must be at least a full letter grade different — for example, a C- to a D-, or a B to an A.

That doesn’t mean Chiefs defense was good; we know it wasn’t. But this defensive data from a single season shows us that it isn’t an absolute requirement to even have a good defense to win a Super Bowl; the New England Patriots’ defensive grades for the year are measurably better than the Chiefs’, but not significantly so — and in no way do they describe a good defense.

So now we have the answer to our first question. What about the second?

We could stop right here, and say that the Chiefs defense only needs to be a little bit better in 2019 to reach — and win — a Super Bowl. Many fans have already reached that conclusion on their own.

The problem is that we’re also facing a lot of speculation about the possibility the Chiefs offense could regress following its incredible 2018 season. What if the defense gets a little better, but the offense gets significantly worse?

So let’s use a statistical system that measures the relative quality of each phase of a team — offense, defense and special teams — and allows for the strength of their opponents. DVOA ratings calculated and published by Football Outsiders do this very thing.

As before, we’ll look at DVOA numbers as letter grades, and mark 2018’s playoff teams in bold.

One thing that should immediately strike you about this data is that DVOA is very good at predicting playoff teams. That wasn’t just true in 2018, either. Over the last 20 years, 78% the top 12 teams in total DVOA during the regular season occupied one of the NFL’s 12 playoff spots.

That’s impressive.

So if you accept that DVOA provides a pretty good picture of a team’s quality, that could lead to a conclusion that the Patriots’ Super Bowl victory in February was a fluke; that row of Cs doesn’t exactly scream championship-caliber team.

We’ll come back to that in a minute.

But first, let’s look at the DVOA grades of the last 20 Super Bowl winners.

At first glance, this data doesn’t show many patterns. But if you look at it for a while, you start to see a few — and the letter grades actually make them a little easier to see.

First, it’s not at all uncommon for teams that are just average on one side of the ball to win a championship. Nine of these 20 teams won with an average defense; 14 won with an average offense. Five (including two Patriots teams) won with an average team on both sides of the ball.

Second, among Super Bowl winners of the last two decades, if one phase of the team is significantly better than another, it’s more likely to be the defense than the offense. Seven teams had significantly better defenses than offenses, while only two had significantly better offenses than defenses.

So how did teams like the 2018 (and 2001) Patriots, 2011 (and 2007) Giants and 2012 Ravens all win championships with average teams?

Part of the answer has to do with the nature of statistics — even advanced ones like DVOA. By definition, statistics describe what happens against all opponents, instead of a specific opponent. So they have limited value in the postseason, when you have to figure out a way to beat that opponent on that day in that stadium.

Another part of the answer has to do with quarterbacks. We think of the quarterback as the most important player on the field — but we also know this is especially true when everything is on the line. It’s not a coincidence that 14 of these games were won by quarterbacks that most anyone would agree were elite: Tom Brady, Peyton Manning, Russell Wilson, Aaron Rodgers, Drew Brees, Ben Roethlisberger and Kurt Warner.

It’s the same with coaches. When facing another good team, game planning and other strategies assume even more importance than they do in regular-season games. It’s no accident that Bill Belichick coached in nine of the games on this list — and won six of them.

---

There was a time when a discussion like this one was almost too depressing to contemplate. “Sure,” we might have said. “The Chiefs look good on paper, but how are they going to beat the Patriots at Foxborough, the Steelers in Pittsburgh — or even win a playoff game at home?”

I understand. We’re long past the point where statistical probabilities are convincing.

So consider this:

A year from now, it’s possible we will see the most important changes to the Chiefs weren’t the new defensive scheme and attitude, or even new players like Frank Clark and Tyrann Mathieu.

While those things will matter — and will be likely be reflected in defensive statistics such as I’ve shown here — the most important factors for postseason success might be much harder to measure.

Patrick Mahomes has clearly demonstrated the ability to be among the league’s elite quarterbacks for years to come. And with each passing season, Andy Reid has knocked over one seemingly-impossible task after another.

Steve Spagnuolo may have an inconsistent record as a defensive coordinator, but he’s done something that most thought impossible: with just an average defense, he shut down the Patriots juggernaut that had won 18 straight games on the way to Super Bowl XLII.

Fans may remember the game for David Tyree’s famous helmet catch, but that wouldn’t have mattered if Spagnuolo hadn’t figured out how to stop that opponent on that day in that stadium — holding the Patriots to seven fewer points than any other team that season.

And joining Spagnuolo is defensive line coach Brendan Daly, who has three Super Bowl rings after serving on Belichick’s staff for five years.

So yes... a measurable (if not significant) improvement in defensive statistics will lead to a better chance the Chiefs will win a championship. But factors we cannot measure may prove to be the most important of all.