From the FanPosts -Joel
Welcome back AP readers. If you read my last FanPost, which is here, or the original FanShot (h/t jayhawksNchiefs) that it was based upon, which is here, then you have an idea of what this FanPost is going to look like. There is one not-so-subtle difference between the two posts, however. This one is going to be huge. Like, mammoth huge. Maybe not quite MNchiefsfan or War and Peace huge, but big. This is not done because I like to hear myself type, or because I get paid by the word. No, this post will be huge for two reasons:
First, I want to spend a bit more time in the post explaining what I am actually trying to show, or why I think the data shows this, whereas last time I decided to "let the data speak." What I found, from many of the comments, was that the data was speaking a language that many folks either didn't understand or didn't want to understand. Therefore, I will include more explanation about the data and about statistics themselves. To help you skip anything that is not directly related to football, I will label any discussions about stats with the header of "Nerd Alert!" Feel free to skip those parts (even though it will break my nerd heart) and go right to the pretty pictures (even though much of the really interesting stuff will be in the nerd alerts (see how I'm pushing the learning aspect here? (Wow, my first triple hyper-paranthetical, MN will be so proud!))), but if you do skip my explanations, and then proceed to comment about about something I clearly address within the body of this post, I reserve the right simply respond to your question or accusation with: "Read the Post."
Second, I am analyzing a set of 6-way split statistics. I am looking at 14 different split statistics. And I am comparing those between 5 different quarterbacks over a 4 year time stretch. This means i'm dealing with over 1600 data points and 480 separate analyses. It just takes a lot of room to convey that many comparisons. Normally, I would simply throw out any of the charts or findings that didn't produce sound results, or that I though were irrelevant to the argument. This sort of judgement, however, has gotten me labeled as a liar, a cheat, or a Smith homer fanboy, and I'm therefore including absolutely everything, no matter how ridiculous or trivial. So, I won't apologize for the length of this FanPost, but you may want to grab your coffee and a blanket.
The Statistical Design
I will begin by explaining why I chose the data that I did, where it came from, and how I have set it up for analysis.
I decided to look specifically at Alex Smith's ability to throw the "long ball" based on some comments from my previous post. Now, I want to emphasize that this post is designed to look at Alex Smith's ability to throw the long ball. That's it. Many of the questions about Alex's Smiths arm came from a discussion about whether or not he could lead the team to big comebacks or post-season victories without throwing a good long ball. This post will not answer that question. It is not designed to test whether or not a good "long ball" leads to wins, whether it's necessary for wins, or if it is a key to off-season success. Those are all very good questions. Other stats can look into those questions, but this post won't. In other words, this will not answer, nor can this answer, whether or not the Chiefs will win games next year. It is about how well Alex Smith throws the long ball.
I chose this topic because I don't honestly know. And I figured that before we argue whether or not Alex Smith's inability to throw long is going to hurt us, i wanted to know whether or not that disability really existed, and how bad it might be. I have chosen to investigate that question in two ways. First, I look at trend data over the years to examine the different types of passes Smith has thrown.
WARNING! - Nerd Alert #1:
- Gee Bull, is it right to look at trend data, or isn't that "cheating" by ignoring his early years, or some trick to get around his average numbers, which don't look very good?
- Excellent question. Now, if you read my earlier response to saints_chiefsfan1979's FanPost, linked here, then you have heard most of this before, but I'll explain it again. NFL stats are what statisticians call "panel data." Panel data comes from measuring the same subjects, all taken together, at regular intervals over time. Scholars often use panel data for presidential polls. Why? A randomly sampled poll can tell me how the country feels about a president at a snapshot in time. Panel data, where I track the same 1000 people over many presidencies, can tell me how people's opinions change over time. I can say definitively, not that two different voters may have come to different conclusions, but that a single voter changed their opinion about a president or a policy. That is the beauty of panel data. It allows you to track trends, which provides more data about "why" questions than random samples over time can. Now, you can ignore that "extra" information if you want to, but it’s there and it's real. My "trend" graphs are simply trying to pull some of that information out of the data.
- "Trend" data is not some tricky way to ignore the past. It is in fact the way we learn from the past. I can’t get trends without knowing the past. The data is not cherry-picked, left out, or deleted. The past is essential. And trends are a valid way to interpret it. They provide more information than simple averages do. I provide an example of two different stocks in my link above, one that is trending up (Let's call it Apple), the other trending down (Let's call it Gateway). If we look at their two graphs over time, we clearly see one is getting better, the other is falling off. If we only look at the overall averages, we miss that information. It's there, we just choose not to look at it. And, if the overall average of "gateway" is higher than "apple," you could be led to think it's a better deal or a stronger stock than apple, when someone with the trend data could clearly see that you are wrong. This is why we create panel data about the stock market. And it's why we keep panel data about the NFL. Trends are not a trick.
END - Nerd Alert #1
The trend data I use to look at Smith originates from the ESPN website here. These are Smith's split stats. I am specifically looking at the section labeled "By Pass Play." This split stat breaks all of Smith's passes into six categories based on the distance he actually threw the ball in the yard. We can then look at his short throws against his long throws without worrying about whether or not a receiver's YAC are skewing our statistics. I use the data from this table to calculate and look at Smith's efficiency stats. These stats include: Completions per Game; Attempt %, which shows what percentage of passes he tried to throw different distances; Completion %; average Yards per Game; Touchdown %, which is the number of TDs thrown per attempt, not per completion; Interception %, which also calculated by attempt; Quarterback Rating; and one more unusual stat in this case, which is Average Yards per Attempt (I'll explain later why I call it unusual).
This data only goes back four years. This is unfortunate for two reasons. First, it doesn't cover all of Smith's career, so it doesn't capture his early, terrible years. This is not a deliberate effort to avoid his past. They simply don't have the split data going back that far, and i have neither the time, the resources, or the inclination, to go through three seasons of film, play-by-play, to generate it. Sorry. Second, four data points (2009, 2010, 2011, 2012) is not very good for capturing a trend line. It becomes very easy to have a single really good or bad season greatly influence our findings. One thing does help us out on that last point, however. Each of our four data points in the trend is a composite stat, not a single observation, so it reduces our variance and makes our trends a bit better. Either way, I have to caveat all of my findings with this disclaimer: The trend data in this FanPost is not as complete as in my last post, does not represent the entire picture of Smith's career, and must be interpreted as such. This is not a deliberate attempt to deceive, but a limit of the data available to me.
I will use my trend charts of these split stats to look for interaction effects that could prove interesting. We will see two charts for each stat. The first breaks the stat into six categories. There is a problem with doing this, however. By breaking our data into so many categories, we lower our sample size in each category and lower our degrees of freedom. in some cases, we end up with only a single pass, TD, or INT, to try to find a "trend." That is torturing the data beyond what is useful. To help correct this, I pooled the data into two groups: passes less than or equal to 20 yards in the air, and passes greater than 20 yards in the air. This reduced the variance of our time series data and allowed for statistically "stronger" results.
WARNING - Nerd Alert #2:
- Whoa, Bull. You've thrown out some weird terms here. What are "split stats," what is an "interaction effect" supposed to be, and what's all this about "composite stats," "degrees of freedom", "torturing," and "pooling" data? It sounds like you're cheating again!
- More great questions, so let's take them in order. Split stats are simply that--stats that are split by another variable. In the NFL we can take a Quarterback's passing stats and split them based on whether they played on Sunday, Monday, or Thursday; split them by Home or Away games; split them by Indoor and Outdoor; or in our case, split them all by length of the pass thrown. We call this "splitting," but what we actually produce is closer to "interacting" these two stats. When I analyze the data, I am asking, "does it matter to Smith's completion percentage, whether or not he played indoors?" or better put "Does the fact that he plays outdoors interact with his completion percentage?" This "interaction effect" can appear to us in two different ways:
- First, our split can interact with our statistical averages. This is a very common finding, and one that we expect to see in our given split. We probably do not expect a quarterback to complete the same percentage of 30 yard throws as he does 5 yard throws. We can visually see this "interaction effect on averages" on our Alex Smith charts when one line is parallel to, but higher than, another line. It's there, but not incredibly interesting.
- Second, our split can interact with our trend. This would be a far more interesting finding, and by the way, one that we can only see if we look at splits and trends (Aiken_Drum's averages will never show this). Let's say our overall stat shows Smith's completion % is improving (which it does). A trend interaction would show that while Smith's short game is getting better, his long game is actually going down hill -- which would prove that a generalized assertion like "Smith is getting better at completing passes" is not entirely accurate. Visually, this would look like two trend lines that have opposite slopes on our Alex Smith charts--one pointing up, the other down. This is more intersting because it reveal to us something we never could have known without this type of analysis.
- Unfortunately, not all of our split stat analysis comes our really clear. This is because of our small sample size. Not only do we only have four years worth of data, but when we split that data six ways, we end up with very small samples in each category. This reduces your degrees of freedom, which means you could run out of power in your data to see anything useful. The best way to think of degrees of freedom is to think back to algebra class: if you needed to solve for X, Y, and Z, you had to have three equations. Three equations yields three degrees of freedom. If you put a fourth variable in there with only 3 degreed of freedom, you're screwed. You can't solve for your variables. I call stretching data too far "torturing" the data. On the charts, you can clearly see how "tortured" our data is. When the curved data lines look like a nice even "sine wave" shape, our data is probably okay (more complex statistical tests are required to tell if it is truly okay, but they are usually only necessary if we see the data "looks" bad). If our curved data lines bounce all over the place or look like a crack smoking monkey with a spire-graph drew them, then we have tortured our data.
- A way to fix this is to pool your data. Instead of each observation getting a point, we group them to help out our analysis. This should be intuitively obvious. It wouldn't make sense to track the passes that were 1 yd, 2 yds, 3 yds, 4 yds, etc. individually -- instead, ESPN grouped them in ten yard increments. The question then becomes, how big should we make these groups? It depends on what we want to know and how much data we have. If I wanted to track your daily habits, I wouldn't need to record your activities every second. Every minute would probably be excessive too. If I only observed you once a day, however, my analysis would be poor. Once an hour gives a good balance between tracing what you're up to, and having the resources to record and process the information. This is why our four seasons work out for us to track trends. Each seasons data point is actually made up of many throws, which are "pooled" together and capture enough info to discern patterns. That being said, NFL data is often not as good as other pooled data, like out economic stocks above. For one, it is less precise and more difficult to measure. More problematic is the fact that it happens less often. There a millions of stock exchanges every second, but Alex Smith only plays 16 games a year. He also threw a limited number of passes over the years. In order to get a better picture, then, i pooled his throws into the +/- 20 yard categories.
END - Nerd Alert #2
I use efficiency stats instead of absolute totals (like total number of touchdowns each season) for a very simple reason: Alex Smith did not play in the same number of games every season. Nor did he throw the same number of passes in every game he played. This fact is not something that I am trying to capture in my analysis. In fact it is what I am trying to avoid, so I can strictly isolate my question: "When Alex Smith is on the field and throwing, how well and how often does he throw the long ball?" This is not trying to gloss over the problem that Alex Smith has staying on the field -- which is a real problem as we see here:
In fact, I believe his track record of incomplete seasons is the single biggest knock against Smith (and one I will probe directly in my next post). If someone could look into a crystal ball and tell me exactly how many games he will play, I would feel much better about telling you how much he can help this team. The problem is, injuries are not a statistically reliable measure. We may think we know when someone is injury prone, or "snake bit," just like we may think we know if someone is "clutch," but attributes like those have proven very inconsistent and difficult to statistically analyze. Yet - none of that has any bearing on what we are looking at today - which is about how well he does when he actually gets on the field to throw.
Alex Smith Findings
** Warning: Due to me not paying close attention when coding this data, I got the years in reverse order for every stat. It was too much work to flip it all around. Instead, we just have to read these charts right-to-left. Therefore, a skinny trend line pointing from the lower-right corner toward the upper-left corner is an "improving" trend (exactly the opposite of my last FanPost, but honestly not that hard to see).
Now, onto the pretty pictures. I first looked at how often Alex threw the long ball:
Now, this graph was very interesting to me. We know from my last FanPost that the one category Smith was trending down in was completions per game. May of us speculated that this was due to a transition to Harbaugh's din-and-dunk offense. This chart suggests otherwise. It's actually the screen passes, behind the line of scrimmage, that reduced. The long throws all actually stay pretty even, which suggests Smith was still chucking it downfield as much as he always had. So, how well did he do when he threw it? Good question:
We see a bit on an interaction effect in Smith's completion percentage. We know his overall completion % trends up. We can see his "over 40" completion % however, has been getting worse. This is such a small number of passes, however, that it doesn't stop our pooled data of all passes >20 yards from trending up, especially when we see that his 21-30 yard completion rate has improved more than any other throw he has made. The average yards per attempt are a mixed bag of results, but I think this is a weird stat to look at here anyhow because it includes yards after the catch (YAC) and is based on attempts, not completions. ESPN included it in their table because it makes perfect sense for their other splits, like Home or Away. But, when your split is to designed to divide passes by how far they went in the air, it doesn't tell us much about how those passes are different from one another if we look at a stat that has YAC in it.
We can, however, use these two sets of charts together to form a hypothesis about why Smith's number of attempts per game has been going down. Now, RamX21 suggested that "Smith's teams starting having success when he started throwing less," dropped his mic, and went home. Now, the real question is, which way did the causality flow in that situation. Did the team get better because he threw less and was only asked to throw when Harbaugh knew it would work, or did he not need to throw it less because his early throws got them the lead and he didn't need to throw it as much? The truth is -- we can never fully unwrap the endogenous nature of this question.
WARNING - Nerd Alert #3:
- Whoa. What the heck is endogeneity?
- Endogeneity occurs when your independent variable affects your dependent variable, but your dependent variable also affects your independent variable. A crude analogy can be seen on an Excel spreadsheet. You may have made this mistake before, when you set the formula in Cell #1 as "Cell #2 +5." Then, you set the formula in Cell #2 as "Cell #1 +10." Excel gets very angry with you because it is caught in an infinite regression, and reports it as an invalid reference. Well, that same sort of mathematical error pops up in stats in problems of circular reference, and it's called endogeneity. You can get around this problem when you include external factors, but that goes way beyond our discussion here. It will suffice to say, it's a tough problem, but not an impossible one.
END - Nerd Alert #3
So, in our case which is it? To me, I would say that the fact that Smith completed a higher percentage of his throws, plus the fact that each throw netted them more yards, adds up to evidence that Smith threw less because he didn't need to throw as often. If my first down pass gets completed, and gets me 7 yards, I don't ask my QB to throw two more times. Now, this is conjecture, not a statistical finding, so feel free to offer an alternate opinion.
Now, how effective were Smith's throws?
Wow. Look at those charts. Remember that monkey with a spire-graph I mentioned? This is it. It shows up here because there are so few TDs and INTs in each category that our 6-split stats on top just offer us no valid reference point. Below, we see some better looking charts. They all confirm what we saw before - Smith's TD% and INT% have improved over the last four years, as has his QB rating for both his long and his short throws (with the possible exception of his QB rating on throws over 40 yards).
Now, Although I don't believe we can learn much about Smith's ability to throw the long ball from absolute numbers of completions, attempts, TDs and INTs, I also want to avoid the appearance of avoiding that information, so I'll include these graphics just as a refernce point:
Alex Smith Comparison
So, this has told us a little bit about the consistency of Alex Smith's trends. His long throws have not suffered at the expense of his improved short game. But, this hasn't really told us if he is any good. To do that, we need to compare him to a couple of accepted baselines. Now, my comparisons will be based upon the visual evidence that these charts provide. I will not statistically analyze them for data outliers, confidence intervals, or precise predicted probabilities. There is simply too much data and this would turn into a novel. This will rely on the old eyeball test, and so I cannot guarantee my findings, but I can highly suggest them. I chose four other quarterbacks for this comparison:
- Tom Brady - Generally accepted as a high level QB, but not considered a "strong arm" QB
- Aaron Rodgers - Our high-level, strong armed QB (Included at saints_chiefsfan1979 request)
- Joe Flacco - A good QB with a strong arm. I include Joe because I think he is a good model for Alex statistically, so it will be interesting to see how their "arms" compare.
- Matt Cassel - Weak QB, weak arm. I choose Matty Nice because people continue to force the comparison.
We will go through these comparisons stat by stat. We won't be able to see any interaction effects anymore, because each category is now on a separate chart. We can, however, now look at where Alex sits in relation to the other QBs. We want to look at his average, his trend, and his consistency. His average will look better than another QB's if his trend line is "higher" up on the chart. His trend will look better if his trend line is "steeper" pointing to the upper-left than another QB's trend line. He will show more consistency if his curved line is closer to his trend line than another QBs (if it is really "wavy" with big peaks and valleys, he has not shown as much consistency as the others).
Completions per Game
We will begin with Completions per Game. Is Alex Smith throwing more or less of these passes in each game? Given that his overall completions per game have decreased, we would expect his split stats to drop also.
**Another quick warning: For some reason my legends have dropped off of the three bottom charts of my 6-way splits. In the pictures with six different charts, the graphs on the bottom are (from left to right) Passes 21-30 yards / Passes 31-40 yards / Passes 41+ yards. The colors remain the same. Sorry about that...
Our six-category split works well until we get into the 31-40 Yard, and 40+ Yard charts, where the data starts to look pretty random. What I notice:
- We see that Smith is trending down, and is well below the average on passes thrown behind the line of scrimmage. This fact may challenge some peoples idea of what Harbaugh was doing with Smith.
- He’s at the bottom of the pile in passes in the 11-20, and 21-30 yard range
- He’s very close to Tom Brady (our other "weak arm" QB) in the 31 and over ranges.
- Looking at the +/-20 Yard split - he’s bottom of the pack in both categories
Again, knowing he had declining numbers anyway, this tells us little new information, but confirms that he does not complete the long ball as much as our strong armed QBs. This made me wonder, was it because he missed, or because he was asked less? Which leads to:
Attempts per game
Clearly, Alex is being asked to throw a lot less than all of our other QBs, in both the long and the short game. But how does does his game divide up? Let’s look:
Again--this stat is telling us what percentage of all of a quarterbacks throws were attempted at a certain range.
This to me was a very telling statistic. The 6-split charts on top don’t help us much, mainly because (if you look) the scale for our long passes is looking at the difference between 1% and 5% of a QB's throws. The medium range throws are what really drive this stat, and we see that more clearly in the +/-20 Yard split.
- Alex Smith and Aaron Rodgers are the only two QBs who have trended up in short throws and trended down in long throws
- Even Matt Cassel was throwing a higher percentage of +20 yard balls - who knew?
- Alex Smith has the lowest percentage of passes over 20 yards
So it seems pretty settled, he didn’t throw the long ball much by anyone’s standard. But, how did he do (comparatively) when he did throw it?
Okay, again, the 6-split charts are a bit of a mess, so we have to be careful about drawing any strong conclusions from them, but we do notice that:
- Alex Smith is right there at the top in all six charts.
- He had to "trend" his way to the top in his short game, but...
- He has always been near the top in his throws over 20 yards (up there with Rodgers). Hmmm....
- TD% doesn’t seem to be driven by "Arm Strength" but by talent. Rodgers and Brady lead in both categories. This shoots some holes in the theory that "weak armed" QBs have a harder time with late game drives or needed big plays.
- Alex is right in the middle (above Cassel and Flacco) in both the short and long throws, but below our elites.
- Smith’s pick percentage has improved in both facets of the game, but he’s at the top in the over 20 yard category.
The one thing our Chiefs team (and Fan Base) will not be able to accept is turnovers. This could be an area of worry, and I’ll look more specifically at it in my next post. But, Interceptions also go into the calculation of QB Rating. So let’s see how bad that stat gets effected by what we have seen so far:
- Smith is right at Flacco level in the LOS and 11-20 range.
- Smith has "trended" to the upper level in his 1-10 yard throws
- Smith is second behind Rodgers in 30-40 yard throws
- Smith is second behind Flacco is over 40 yard throws
- Aaron Rodgers is the frickin’ man. Period. We should get that guy.
So Smith completed the ball, but did he move the ball down the field?
Average Yards per Attempt / Average Air Yards Per Attempt
I first looked at the ESPN stat for "average yards per attempt," but again I think this is a bit goofy. It includes YAC in a post where we are trying to ignore YAC in our evaluations. Because of this, I also included the "average air yards per attempt" from the guys at Advanced NFL stats. This stat shows us how far each QB threw the ball, in the air only, on average, given every throw they made in the season. What I see:
- Smith is again in the top half
- Smith is right near Flacco in the short game and AAYPA, but well above him in the +20 yard range.
- This stat - Like TD% - seems to be driven more by talent than arm strength
So, in terms of actually throwing the long ball, completing his throws, scoring TDs, and affecting the team by moving the ball, Alex Smith is all right in the long game. He is not elite, but these charts show that his long game actually appears more efficient than his short game, which has really improved over the years. Arm talent certainly does matter, but not quite as much as I would have thought before I did this analysis.
Some Additional Thoughts
I discussed above why I thought absolute numbers did not matter much in a discussion about QB efficiency. I also said that they absolutely do matter if we want to know how much a QB is going to help his team throughout the season. That question will be the focus of my next post, but because I have some of the data already, I included some charts below that show where Smith stands in terms of total numbers against our four other QBs.
These are more informative than our bar graphs of just Smith's numbers above because we have something to measure them against - some context to put them in. As a reference, I include a small chart that shows how many games each QB played in each year, so you can get a feel for why numbers may vary like they do:
- Alex NEEDS to stay on the field. We can’t afford another Brodie Croyle roller-coaster at QB.
- Even though he played in far fewer games, Smith threw almost as many short yardage TDs as Flacco did in every season.
Well, I hope this was educational. I didn’t learn as much from these numbers as I perhaps would have hoped. Again, I took this on because some specific critiques of Smith made me curious. In the end, it didn’t really strengthen my opinion of Smith, or take much away from it. Perhaps the most interesting thing (for me) was that the idea of "arm strength" or "arm talent" may be overhyped in the NFL.
I had fun putting it together. I know the length of the post was a bit much, but it was a lot to get through, and I thought it would be better if I headed off some questions before they came up.
Last, in case anyone is interested, here is my data set that I used for all of the charts. Feel free to use it for your own analysis if you want to counter any of my conclusions.
Cheers. GO CHIEFS!!