ThThere are a good number of reasons that Kyle Orton isn't really a Super Bowl quarterback. His accuracy is somewhat inconsistent, his mobility and pocket awareness range from mediocre to atrocious, he locks in on Brandon Lloyd, and his arm-strength is fairly average. One reason that I hear the most around these parts has nothing to do with any of these shortcomings. Basically, it's the accusation that Orton plays crappy at important times when the game is in the balance. This is apparently some kind of habitual shortcoming related to either his specific skills or his weakness as a person. Despite pretty terrible clutch play last season and on Monday night, this assertion isn't backed up by the evidence. Specifically, there is really no such skill as "clutch." Players that are better are better when the game is on the line. A quarterback's ability to outperform his normal production level during high-leverage situations is just statistical noise.
Let's define some terms. This post derives heavily from the data collection and analysis done by Brian Burke at Advanced NFL Stats. ANS is built essentially on two key stats: Estimated Points Added (EPA) and Win Probability Added (WPA). EPA assigns each down, distance, and field position situation a number of points that an average offense would be expected to score. Each play either increases or decreases this expectation. The credit or blame for each play is assigned to every player involved.
Win Probability Added is essentially the same stat, except instead of the number of points, the calculation figures out how likely each team is to win the game. The only real difference between the two stats is that WPA takes into account the leverage of the situation. Leverage is basically the odds that a given play will impact a game. So a touchdown scored when a team is down 14 with 2 minutes to go has very little WPA, because the situation is so low leverage, even though the EPA would be exactly the same as a game winning touchdown as the clock runs out.
Although there are some statistical issues with both of these stats and the way Burke calculates them (such as credit apportionment), they generally pass the smell test. For example, Kyle Orton had a WPA of -.11 (meaning he decreased the Broncos' chances of winning by 11%) on Monday night. That is down entirely to the first half interception (-.09) and the fumble in Raiders' territory (-.20). By contrast, the entire late touchdown drive was worth just +.14. That jives with everybody's understanding that Orton lost the game with some stupid plays. Note also that the average QB WPA is slightly above average because of the way Burke's model handles runs (which are generally seen as more negative) and passes. So Orton's -.11 for the game is really, really bad.
I vastly prefer using WPA as a "clutch" metric to something like "game within 7 in the 4th quarter." That's because it really is possible to have massively important plays in the first half, or even in the first quarter.
The real question though is whether Orton has in the past, and will continue to, perform poorly in high leverage (or clutch) situations. The answer is he was atrocious last year. By comparing WPA (which includes leverage) to EPA (which doesn't) we can find out who is helping put their team in more winning positions than their underlying individual stats indicate. Here is a chart comparing WPA to EPA from 2010:
The line indicates "average" ability to translate EPA into WPA. The guy who exceeded his WPA expectation by the most was Matt Ryan, on the top. Orton was the 5th worst in the league by this metric. He is way below the line. But what if we expand it out a little bit more? Here is the same chart from 2009:
This time Orton comes in somewhat above expectation. His performance was essentially the same (about 50 EPA, despite playing more games in 2009) but he went from .09 to 1.6 WPA. Did he just get a lot worse at performing well in high leverage situations? That seems like an odd explanation. The better question for me is really whether a player can consistently produce more WPA than his EPA, yards per attempt, and DVOA would project. In other words, is clutch an inherent and persistent skill?
The most basic way to measure this is simply to see if you can use year one clutch numbers to project year two clutch numbers. We already saw that would have been a fool's errand with Kyle Orton between 2009 and 2010, since he went from comfortably above average to absolutely terrible. Other players who suffered dramatic drops in clutch performance were Philip Rivers (who went from comfortably above average to third worst in the NFL) and Peyton Manning (who went from best in the NFL to bottom third). On the opposite side were Matt Ryan (who went from underperforming in high leverage situations to #1 in the league) and Marc Sanchez (who went from terrible to top 10). An r-squared analysis showed that one year's "clutch" performance doesn't explain the next year's: the r-squared value was .068 (heh, ueber nerdy).
Basically, I think playing well in high-leverage or "clutch" situations is just a subset of being good at quarterbacking. To the extent that some players perform much better in those situations for a year or two, it doesn't betray an ability to be clutch, just a small sample size. Some really great players like Rivers, Manning and Brady have had awful years in terms of WPA that is lower than overall performance, but they balance out. That's because "clutch" isn't really a skill.
Ultimately I'd anticipate Kyle Orton's play in clutch situations to regress back toward his overall level of play as measured by EPA, DVOA and Yards Per Attempt, which have been around average or slightly above average the past two years. This isn't supposed to be an argument strongly in favor of Orton: nobody wants an average quarterback. I just think the "clutch" argument is bunk.