I started writing for TwoQBs last March, mainly about quarterback projections. In April, the draft was coming up and I was talking to my fiancée about how I was spending all my free time. She asked how I was going to project the rookie quarterbacks, noting it must be difficult to translate college stats to pro numbers (she has no idea how right she was). I didn’t have a good answer at the time. To make my projections complete, I leaned on the projections of others as a guide, while checking how rookie quarterbacks have historically performed.
For the past 10 months or so, it’s been in the back of my mind. My new data science and coding skills have helped me come up with a plan of action. After collecting players’ college data, I set out to evaluate quarterback prospects in an objective manner. I tested a few popular metrics to see how they correlated with draft position and NFL success. I tested these stats for QB performance, not necessarily fantasy performance. But predicting/evaluating offensive efficiency in general is still important to fantasy research. Volume is king, and an offense’s ability to sustain drives and get near the end zone can greatly influence volume.
Refresher on Popular QB Efficiency Metrics
Yards per Attempt (Y/A) is the simplest passing efficiency metric. It’s easy to understand, easy to calculate, and is visually apparent on both a play-by-play basis and when looking at a box score. It is also the most stable metric of the bunch year over year.
Adjusted Yards per Attempts (AY/A) stems from Y/A, accounting for touchdowns and interceptions, and putting them into a single number on a similar scale to Y/A. Josh Hermsmeyer has shown that passer rating is essentially a convoluted version of AY/A, and AY/A is more predictive. AY/A is also more descriptive as an all-encompassing throwing metric than Y/A. The downside? AY/A loses stability year-to-year versus Y/A, as touchdown rates carry a ton of variance.
Adjusted Net Yards per Attempt (ANY/A) takes everything a step further. By including sacks as pass attempts and subtracting for yards lost on sacks, ANY/A is an efficiency metric accounting for the entire passing game. You might think this is biased towards quarterbacks with good offensive lines. However, some smart people have shown that sack rate is inherently tied to the QB himself, and then influenced afterwards by the rest of the offense. Back in 2009, Jason Lisk showed that when quarterbacks switch teams, their sack rate is actually the most likely stat to remain constant.
Adding sacks to the equation brings back most of the stability lost when we included touchdowns with AY/A. While Y/A is still a bit more stable, I like ANY/A because it is comparable in stability while accounting for more context. Typically, the more complicated a formula is, the more we lose stability due to the mess of relationships between variables. So while we always want to look for metrics with more stability and predictive-ness, those that account for more context get a break from me as long as they’re in the ballpark.
Besides messy passer rating formulas, the above metrics are most popular for measuring quarterback efficiency. Josh Hermsmeyer is challenging that with PACR, which uses Air Yards instead of pass attempts as the volume component. Air Yards matter. I wish we had comprehensive data for the stat on college players, and while Josh continues to spread the gospel on Air Yards’ effects on NFL passers and receivers, we have to make due with what’s available at the college level.
So I pulled all the data I could on college players from Sports Reference. The resulting dataset included all offensive statistics for FBS players since 2000 on a game-by-game basis. Afterward, I started testing the above metrics until I got to ANY/A. Sacks and yards lost on sacks are not recorded in the same way at the college level as they are in the pros. As a result, I could not calculate ANY/A, which disappointed me. I could measure how efficiently the quarterback throws the ball, but the goal of my research was to measure passing performance in an all-encompassing manner. Luckily, I realized sacks are recorded as rushes for negative yards for the college quarterbacks.
Naturally, I decided to substitute the quarterback’s rushing stats where we normally include sack stats in the formula for ANY/A. This allowed me to include sack numbers I didn’t have access to. As a consequence, I had to include all rushing stats, which I ultimately decided was a positive thing rather than a negative. Scrambles are an important part of the passing game in my eyes. My new formula also included designed runs, which I was wary of since those play calls drop significantly in volume in the NFL. For instance, Cam Newton has averaged 7.6 rushes per game in his pro career, which is an all-time high for an NFL quarterback with at least 1,000 pass attempts. No one is even close unless you limit Michael Vick’s numbers to his starting years in Atlanta (and he would still rank behind Cam). However, Newton carried the ball (including sacks) an insane 18.9 times per game during his Heisman-winning 2010 campaign at Auburn.
All this is to emphasize that I was worried college quarterbacks’ rushing volume would be over-weighted and skew the stat. However, after testing the numbers at both the pro and college levels, I was pleasantly surprised with the results. The metric I produced is Total Adjusted Net Yards per Attempt, or TANY/A. For NFL quarterbacks, it is simply ANY/A with rushing stats added. For college quarterbacks, it is AY/A with rushing stats added (which already include sack stats).
When I realized this was somewhat predictive, I was excited. I had found a new quarterback efficiency metric to significantly improve my QB evaluations. Except it isn’t exactly new. A few days later, I stumbled on Bryan Frye’s TAY/P, or Total Adjusted Yards per Play which was similar to TANY/A, but adds in a few other things. I emailed him and he enlightened me on the history of football analysts stumbling on a metric very similar to each of ours.
A Lesson to be Learned
Bryan developed TAY/P in 2012 on his old site, published a refined version on Football Perspective in 2015, and has written about it there since. Kevin Kolbe published an article for Football Outsiders in 2016 with a metric he called TANYA. There’s a post on Field Gulls by the user “betaparticle” from 2014 with a similar metric also called TANYA. In 2011, Neil Paine wrote an article about his “Ultimate Adjusted Yards Per Attempt” for the PFR blog. Going way back, respected historian and stats maven David Neft published an article for The Coffin Corner in 1993 with a very similar metric to all of these.
I was a bit bummed my invention was really a rehashing. Thinking back, it’s entirely possible I had seen TAY/P at some point, forgot about it, and it was somewhere in the back of my mind helping develop my version. Not only was I beat to the punch, I was beat by five people (at least), and one of them got me by a quarter-century. Some of you reading this may have seen or used Bryan’s TAY/P before, and others may see TANY/A as brand new. While I believe my scope, analysis, insight, and future projects regarding TANY/A will all be fresh and unique, I do not want to take any credit that deserves to go to the guys listed above.
Total Adjusted Net Yards per Attempt is a bit wordy and the formula is long, but it is quite simple and intuitive. Breaking it down to its elements, ‘Yards per Attempt’ is straightforward. ‘Net’ means we’re including sacks. ‘Adjusted’ means we’re including touchdowns and interceptions, adjusting them to a scale that fits yards. ‘Total’ means we’re including rushing stats. If that doesn’t clear it up for you, hopefully this does. TANY/A simply measures yards per play when a quarterback is involved in some way, accounting for touchdowns and interceptions.
The results are not groundbreaking by any means, but I believe this adds another level to the foundation of existing QB efficiency measurements. At the NFL level, TANY/A is a bit less stable than Y/A, but more stable than the other metrics. It is a bit better than ANY/A, which I was happy about since it adds in a new variable that could have easily thrown off the stability (rushing efficiency). Y/A is still the most stable, but TANY/A is the best for looking at all-around passing efficiency. While TANY/A takes a bit more work, it can still be calculated in your head from a box score, which is important to me personally.
I calculated TANY/A for all college quarterbacks to play in the FBS since 2000. I then looked at all QBs drafted in that timeframe (technically from 2004 on, to get full college stats for these players). I tested all the above metrics besides ANY/A (which is unavailable in my college data) to see which ones correlated with draft position best. I used draft capital instead of draft position, calculating draft capital using a log-based system, which approximates the charts NFL teams use for trading draft picks.
Career Y/A in college was somewhat predictive of draft capital spent on the quarterback, AY/A was a bit better, and TANY/A was even better than that. But while TANY/A is predictive enough to be included in a model, it is not predictive enough to estimate draft position on its own.
After seeing these results, I messaged Josh Hermsmeyer and gave him a summary. After getting his approval, I finally allowed myself to be excited and dive deeper into my study.
Comparing TANY/A to Other Metrics
I have charted the yearly marks of QB efficiency metrics since 1981, when sacks became an official statistic. I included Y/A, AY/A, ANY/A, and TANY/A (raw, not schedule- or era-adjusted) to get an idea of how the scales and trends of each compare.
Note how AY/R (yards per rush with touchdown bonus) is also included for non-QBs to show the difference between the overall effectiveness of quarterbacks and the traditional rushing game. Here are my takeaways:
- Y/A today vs. 1981 is similar, with a bit of variance in between.
- AY/A has trended upward significantly since 2003, thanks to an increase in TD/INT ratio.
- INT% has consistently declined, dropping from 4.3% in 1981 to 2.4% in 2017.
- TD% looks similar to the Y/A line, so the drop in interceptions is most responsible for the increase in passing efficiency.
- ANY/A has followed a similar trend as AY/A, but the numbers are dragged down by inclusion of sacks.
- Sack% has been relatively consistent, but yards lost per sack have trended down since 1981.
- This doesn’t change the overall efficiency numbers much, but combined with the trend in INT%, it’s pretty obvious QBs are getting the ball out of their hands quicker.
- TANY/A is almost equal to ANY/A in every single year, which is very interesting to me.
- Quarterback rushing efficiency is a bit better than that of non-QBs, but it’s not quite as efficient as passing (even when sacks are accounted for).
- Since QB rushing attempts account for six to nine percent of all QB plays, this barely drags TANY/A below ANY/A.
That last point may make you say, “What’s the point of TANY/A if we can just look at ANY/A and get basically the same outcome”? For starters, TANY/A is slightly more stable year-to-year. Also, TANY/A data is much more available than ANY/A for historical college QBs, since sacks are not broken out from rushing in most datasets. So we can compare college efficiency to NFL efficiency, apples to apples. TANY/A also allows us to reward signal-callers who excel in rushing, but not in a dramatic fashion.
Improving on TANY/A
My first task was accounting for strength of schedule. Josh has shown how adjusting for schedule can often be a fool’s errand at the NFL level. It can help contextualize, but it often takes a lot of effort and adds little-to-no gain. However, college football is a different animal. Schedules are far more varied, conferences are about as balanced as talent on NBA rosters, and each team plays (at best) 10 percent of the entire field. I theorized that strength of schedule was meaningful in college, and it was time to put that to the test.
Since I had game-by-game data for the college players, it was relatively easy (but tedious) to account for schedule. I calculated the average TANY/A against for each school by year. For instance, the 2011 Alabama team allowed a 2.43 TANY/A against, while the 2013 Eastern Michigan team had a 9.33 TANY/A against. The FBS average is 5.85, while FCS defenses give up 7.67 TANY/A to FBS teams.
For each player, I calculated their career strength of schedule by averaging the TANY/A against for all their opponents. I weighted it for the amount of opportunities the player had in each game, so as not to overweight games in which they barely played. The result was TANY/A*, which adjusts TANY/A based on career strength of schedule.
The correlation for TANY/A* (0.180 r^2) and draft capital was significantly better than TANY/A (0.141 r^2), and nearly 50 percent better than the other established efficiency metrics (0.125 r^2 or lower). This seemingly confirms that strength of schedule is an important factor when measuring college performance. But I also wonder whether accounting for strength of schedule unintentionally acted as some sort of proxy for the level of conference or school where the quarterback played. Some mathematical models for evaluating prospects will reward passers in the Power 5 conferences, for example. Adjusting for schedule improved the predictive-ness of my metric, so I’m happy either way. That correlation is still relatively low, and it goes to show that college performance is a small piece of the draft evaluation puzzle for quarterbacks. Something we knew already, but it’s confirmed by the numbers.
Looking at the data, a model based only on career TANY/A* in college plus draft position can do a good job of predicting NFL success. At the very least, it does as good a job as can be done when predicting how 22-year-old prospects will progress with just two objective measures. Including age, data for measurables (height, weight, combine metrics), and further contextualizing TANY/A* will hopefully improve the model considerably, and I’ll have more on that by April.
One feature of TANY/A* is how it contextualizes performance for both schedule and era. As shown above, this improves the predictive-ness, but it also means we can compare quarterbacks from different eras meaningfully. On top of any predictive work I do (especially with QB prospects), I aim to do some fun projects like historical rankings. Feel free to send me ideas on twitter!