Never Tell Me the Odds: The Strange Example of Pulaski Academy (Amended)

Don't forget Draftivus: http://www.milehighreport.com/2010/4/8/1410543/announcing-the-2010-draftivus-7

    "C3PO: Sir, the possibility of successfully navigating an asteroid field is approximately 3,720 to 1."
   
    "Han Solo: Never tell me the odds."

    This classic scene from the movie, The Empire Strikes Back is a wonderful example of the quandary faced by every NFL coach in just about every game.  In a game wherein there are around 3700 different factors to take into consideration, there are times when the odds and statistics advocate the coach doing one thing while the coach may be making a choice based on experience, game-moment observation, or just a plain, old, "gut" feeling.  The game of NFL football is like an intriguing blend of gladiatorial games played by championship poker players using the strategies of chess.

    Steve Nichols shared an article he read last year with TJ Johnson and Brian Shrout.  It was an article about The Pulaski Academy Bruins, a high school football team in Little Rock Arkansas.   The Bruins have an odd approach to football: they "go for it" on every fourth down.  Steve shared that, as crazy as that sounded, the article had statistics to support the view that while attempting to convert on every fourth down may lead to some bad field position, more often than not the offense was going to convert the attempt into a first down.  Steve was also surprised to discover that the Pulaski Academy Bruins list neither a punter nor a kicker on their team.   Based on his coaching background, Steve disagreed that going for it on every fourth down and short yardage conversion was the right call.

    Steve invited TJ and Brian to participate in a discussion of this topic.  So, we gathered at the Lone Star Steakhouse and Saloon in Long Beach, California via wireless internet, email, and extensive use of Brian's imagination -- just kidding.  We'll share our discoveries after the fold.

http://www.dennismansfield.com/business/2009/09/going-for-4th-downevery-time-every-gamethe-pulaski-academy-bruins.html

    Coach Kevin Kelley adopted a philosophy for his football team -- they never, ever kick the ball on fourth down.  They do not even run a return when the other team kicks.  Coach Kelley believes that, statistically, his way is better.  He backs his philosophy up by the simply expedience of winning.  He does not list a punter nor a place kicker on his roster.  He does not punt nor attempt field goals on fourth down.  Kick offs are either on-side attempts or deliberately kicked out of bounds.  Returns are not handled, instead the Bruins allow opponents to down the ball.  Coach Kelley isn't certifiably insane, he bases his decision on cold, rational numbers.  When asked to comment on Kelley's approach, TJ's first reaction was:

Although it's difficult to apply statistical analysis to a specific team at the high school level without having their numbers in front of me, it's likely that Coach Kelley has some of his numbers that back up his strategy.  Imagine a situation at the high school level, in which my punter and punt-coverage unit might be giving the other team at mid-field, or worse.  The conversion percentage I would need to justify going for it on 4th down would be 30-40%.  Given these kind of percentages, Coach Kelley is usually making the right call.  Each situation would be different, but generally speaking, aggressive play on 4th down, specifically 4th-and-short, is the right move.

(photo courtesy of the Pulaski Academy website)

    As mentioned above, Coach Kelley has not called a punt nor a normal kick-off since 2007.  He has determined that the average high school punt play will push the opponent back an average of 30 yards.  He has also calculated that his Bruins convert approximately 50% of the time.  Kelley also cites the following data: On a typical 4th and long play near one's own end zone, the opposing team will take possession inside the punting team's 40 yard line and then move down to score a touchdown 77% of the time.  If the offense goes for it and misses, giving the opposing team the ball inside the 10 yard line, the opposing team will score a touchdown 92% of the time.  According to Kelley, his team only increases the opposing team's chances of scoring a touchdown by 15% by giving his offense an extra chance of staying on the field.  He has similar numbers to back up his decision to always try an onside kick.  His data explains that the typical kick-off gives the opposing team the ball at it's own 33 yard line, while a failed on-side attempt gives the opposing team the ball at their own 48.  His Bruins have been able to recover approximately 25% of their onside attempts.  Kelley believes it is worth the risk of giving up 15 yards for a 1-in-4 chance to put his offense back on the field after a score.  Steve was somewhat skeptical about this approach, based on his own experiences as a coach.


Going For It on 4th Down -- A Coach's Perspective

    The discussion was begun by Steve sharing some thoughts based on his experiences as a coach:

I never coached offense; I always coached on the defensive side of the ball.  If our HC missed a game, our OC would have called the offense.  If both the HC and the OC went missing, the Speaker of the House or the Secretary of Agriculture would have assumed the play-calling duties before I would have.  I'm technically proficient enough to run an offense, but I never had a need to study the offensive playbook.  I could have stepped in and assumed the HC's job for a time, but even in that role I would have never called an offensive play.  On the other hand, I was given pretty much sovereign reign over the defense.  The HC spent almost all of his time with the offense, and I had the defense to myself.

However, I will say this: I would never, ever go for it on fourth down on every fourth down play.  Even as someone who considers rational logic over emotion and tradition, I would ignore the logic if it told me to go for it.

TJ provided us with an additional rationale for Steve's position:

We know statistically that coaches in the NFL punt the ball away and kick the field goal on 4th down way too often.  I've used this quote before, but it's worth repeating again, from the groundbreaking book, The Hidden Game of Football:

"The idea that you-gotta-come-away with something is on page six of The Coach's Book of Conventional Wisdom, which every coach receives along with his first whistle on a lanyard.  It's in red ink.  Underneath, it's explained that any team that gets close to the goal line and doesn't come away with a point or so will undergo a psychic shock roughly akin to being weaned.  So he [coach] ignores his offense, which has been doing a good job, ignores the fact that even the best field goal in the world is still worth less than half a touchdown, ignores the fact that the following kickoff will give the opponent halfway decent field position, and sends in his kicker.  So for a few minutes, he has a 3-point lead . . . . "  (p. 152)

    Even among the coaches who are considered to be "innovative," conservatism reigns.  In 2008, Miami Dolphins head coach Tony Sparano instituted a variation of the wildcat formation that met with great success.  Even so, the Dolphins only attempted to convert a 4th down 15 times during the year, they attempted 25 FGs, and punted 74 times.  So, out of a total of 114 4th down opportunities, Miami only dared a 4th down conversion 13% of the time.  Steve and TJ did acknowledge that while the conventional wisdom which forms the core of most coaches' thinking tends to be skeptical of the approach advocated by Coach Kelley, there are times when choosing to attempt a 4th down conversion make sense.

Obviously, there are times to go for a fourth down conversion.  It may be a desperation move (time is running out and the team is losing).  It may be a move based on strength (the opponents are worn down and puking on the field, and our RB is the best in the state).  It may be based on deception (we send out our punting unit, and then rush into a goal line formation).  It could even be based on motivation (our guys need the morale boost right now, or we need to stab the other team in the gut).  There could be many reasons to try and convert a fourth down.  -- Steve

In high school, where punters and field goal kickers are much less skilled and productive than in the NFL, the need to go for it on 4th down is probably even more magnified.  I would be hard pressed to tell a high-school coach to roll out his punter on any 4th-and-5 situation, when his punter probably averages around 25-35 yards per kick.  That's because the Expected Points Value for the opponent when they receive the ball will be similar to if the opponent receives the ball on a turnover or downs.  And the chance of making that 4th down would be high enough to justify taking the risk.  -- TJ

    But still, a sense of skepticism remains about the Pulaski philosophy remains.  Steve provided a compelling case for why a coach should not go for it on each and every fourth down:

But there are many reasons not to.

The stat may be based on "all things being equal . . . "
If a stat tells me a QB should do a certain thing in a certain situation, this may be advisable.  However, as a coach, I also want to look at factors outside the stats.  I might look at the quality of my opponent's cornerbacks versus the quality of my receivers before calling a certain pass play.  I might also look at the score, injuries, fatigue, momentum, psychology, etc.

In fairness to TJ, one can always create a ridiculous scenario that runs over the stats.  My point is only that things are never "all things being equal" in a game.  there are variables at the start of the game, and variables that change throughout the game.

In the case of always going for it on fourth down, even if we limit this to one or two yards, we are taking away too many factors.  We are also denying ourselves a tool, one that we can use to back up our opponents.  The game starts with variables that aren't equal, and those variables move up and down the scale as the game progresses, creating even more chaos.  The coach needs to have a strong grasp for the odds (the stats) in his decision making; I'm not making an argument that stas are a weak tool.  What I am saying is that we have to keep in mind that today's issue is forcing an absolute on us, that we should do something every time.  Those situations are rare in football.

I'd like to use the poker analogy.  The stats may tell us that we have a good hand or a bad hand.  But NO accomplished poker player is always going to bet his hand based on the stats every time.  Part of the game of poker is fooling opposition, and part of the game is not being fooled by your opponent.  Spock would be a terrible poker player.  He would always know the odds, but his opponents would always know when to raise and when to fold.  That's why Kirk is the Captain.

In sum, my first reason is, "As a coach, I don't want to be confined in my decision tree.  I want my choices narrowed (for the sake of game speed), but I don't want to feel my play calling to be restricted by an absolute rule.

    This is a very important point.  It is important for a coach to game plan and play call with a fair amount of flexibility.   This would be done to keep the opponent off-balance, if nothing else.  As Steve mentioned using the analogy of Star Trek's Mr. Spock as a poker player, if the opponent can anticipate what a team is going to do, it has increased it's chances of stopping the offense.  If the opposing team knows your offense will go for it on every fourth down, it can better plan to stop you.  Yet, even knowing that Pulaski is going to go for it, Coach Kelley has found his team being fairly successful.  So, why don't more teams go for it more?  Steve offers another reason:

If this were a good idea, why don't more coaches do it?
I'll admit something here.  This argument isn't fair either.  It falls under the fallacy of "appeal to authority."  In other words, just because 99.9% of coaches won't go for it on 4th down, doesn't mean that the idea is wrong.  99.9% of coaches could be wrong.

However, it is worth mentioning that coaches (from HS up to the pros) are paid to win games, and dismissed if they can't perform.  It follows that coaches would follow up on something that would give their teams an edge.

In sum, my second point is, "Coaches learn the game from other coaches.  Through 'survival of the fittest' we end up with the systems and plays we have today.  If an innovative idea comes along, and smashes tradition (like the west coast offense), it will gain followers.  Most coaches don't experiment though.  We follow what has always worked."

This may sound like a terrible thing, "We do it because that is how it has always been done."  However, there is an argument to be made for tradition.  Most innovations do not work; they fail.  I don't care if I'm an innovative coach or not.  My job is to win games, and I'll study anyone from Sun Tzu to Lombardi to steal their ideas.  Learning from the past is a virtue too.

There are a great many coaches I admire and look up to.  None of them go for it on fourth and short every play.  Who am I to risk my job, and my players' happiness, on a tactic that my heroes have (surely) looked into and rejected?

    TJ suggested additional reasons why more coaches don't go for it more often on fourth down:

Which brings us back to the psychology of coaching.  The vast majority of coaches are not like Coach Kelley, or Bill Belichick.  Most are conservative on 4th down.  Why do coaches, who presumably want to maximize their team's point production, play it so safe on 4th down?  James Surowiecki, author of the book, Wisdom of Crowds, suggests several reasons:
1. It's easier for coaches to justify the way things are than to justify how things could be different.
2. The coaching profession is not very diverse.  Therefore, the profession is less likely to come up with             radical innovations, and even less likely to embrace them.  To quote Surowiecki, "the errors most football coaches make are correlated: they all point in the same direction."
3. Coaches are averse to risk.  Again to quote Surowiecki: "Sticking with the crowd and failing small, rather than trying to innovate and failing big, makes not just emotional but also professional sense.  This is the phenomenon that's sometimes called herding.  Just as water buffalo will herd together in the face of a lion, football coaches, money managers, and corporate executives often find the safety of numbers alluring -- as the old slogan 'No one ever got fired for buying IBM' suggests."

This last point suggests that the other thing that Coach Kelley and Bill Belichick share in common is job security.

Going for It on 4th Down -- What do the Statistics Say?

(photo courtesy of the Pulaski Academy website)


    Thus far, we have focused mainly upon why a coach may, or may not, choose to go for it on fourth down.  It would be helpful to add in the perspective of statistics.  This involves two basic concepts: risk versus reward, and statistical analysis.

What do the stats actually say?
TJ may find that the stats don't support today's issue.  For example, it may be a simple thing to say that, "The odds favor conversion on fourth and one."  But that isn't enough, at least for me.

For example, if I fail to convert on 4th and 1 on my own 15 yard line, I am giving my opponent a (pretty much assured) 3 point field goal.  I am also giving my opponent a reasonable shot at a touchdown.  The odds may be for conversion, but do the odds give me enough of a reward to compensate for the risk?

Here's an example of risk versus reward.  What if I told you that I had four cards, and if you pick the winning card you get $10.  The cost to play is $1.  You would be correct to play the game.  You have a 25% chance of winning, so you could play four times, lose 3 out of those four times, spend $4 and make $10.  However, what if the game costs $5 to play?  After four tries you win $10, but lose $20.  It isn't worth it.  

Lotteries and Las Vegas prey on folks using odds.  For example, you spend a lot of money for a minimal chance of winning more money whenever you hit the tables in Vegas.  In the lottery, you might spend a dollar for a chance at a million dollars, but if the odds of winning are 5 million to 1, you will end up spending 5 million dollars to win 1 million dollars.  It is a terrible bet.  (A stats prof once told me, in defense of Vegas and lotteries, that folks are spending their money for a dream, not actual compensation.

An actual statistical breakdown by TJ may reveal that a team's chances are better than even on 4th and 1 (I believe that they are too), but that the game circumstances don't allow for a good return for the risk in most cases.  -- Steve

    Steve brings up a very good point here.  What coach wants to increase the likelihood that his opponent will be able to score additional points?  Kelley's point of view, as mentioned above, is that when an opposing team starts at the punting team's 40 yards line, the opposing team will score a touchdown 77% of the time.  When the opposing team gets the ball inside the 10 yard line from a failed 4th down attempt, the opponent moves on to score a touchdown 92% of the time.  Kelley believes -- based on the fact that his team converts approximately 50% of their attempts -- that a 15% increase in the risk of their opponent scoring a touchdown is not sufficient to override the 50% chance of keeping his offense on the field.

    As Steve also mentioned, it may be that a statistical analysis would advocate that a team go for it on 4th down.  TJ applied his EPV analysis tool to an instance in which an NFL team went for it on fourth down.

We can easily demonstrate the "correctness" of this type of decision in the NFL.  In fact, let's use the most over-analyzed and controversial decision from the 2009 season to demonstrate: Bill Belichick's decision to go for it from his own 28 yard line against the Colts in Week 10.

As you may recall, the New England Patriots held a 34-28 lead with 2:08 remaining in the game and they faced a 4th-and-2 from their own 28 yard line.  At the time, Belichick was trying to get past the two-minute warning so that he could run out the clock and keep Peyton Manning off the field.  But to make this happen, Belichick was acting as any rational coach would, trying to maximize his points value and win the game.  So our Expected Points Value (EPV) would apply to this situation.

Let's say the Patriots gained the two yards needed for the first down.  Their new EPV would have been 0.48 (1st and 10 at their own 30 yard line).  However, if they only gained 1 yard, they would have turned the ball over, and the Colts would have been facing a 1st-and-10 at the New England 29-yard line.  The EPV (to the Patriots) would have been -3.51.

And what would these values have been if Belichick had decided to punt?  Well, we can make some assumptions.  First, New England's punter during the year, Chris Hanson, was netting about 34 yards per punt in 2009.  Thus, Belichick could reasonably assume that, had he punted the ball, the Colts would have likely faced something like a 1st-and-10 at their own 38-yard line.  The EPV to the Patriots of this down and distance would have been -1.048.

Now we have all of the information we need to determine if Belichick's decision was the correct one.  Using the probability equation:

(p)(0.48)+(1-p)(-3.51)=(-1.048), where:

    p              =     needed probability of converting the 1st down
    0.48         =     EPV of 1st-and-10 at the 30-yard line for the Patriots ***
    -3.51        =    EPV of turning the ball over and having the Colts with a 1st-and-10 at the Pats' 29-yard line
    -1.048      =    EPV of punting with the Colts facing a 1st-and-10 at their own 38-yard line

We solve for the variable "p."  The answer is 60%.  In this case, if Belichick thought he had a 60% chance or better of getting the 1st down, he should have gone for it.  Of course, this is exactly what he did, given the Patriots' 4th-down conversion percentage facing 4th and 2 yards or less was 73% for the year.  Although it didn't work for Belichick on that one specific attempt, it doesn't mean the strategy was statistically flawed.

    A preliminary response to TJ's analysis would suggest that the statistics advocate that going for it on 4th down is a good idea -- at least in the case of the New England Patriots in 2009, when the situation is 4th down and 2 yards or less to go.  Whether or not this could be generalized to other 4th down situations for the Patriots and/or other NFL teams would require additional research.  

    One final observation that has an impact on why a coach might choose to go for it on fourth down comes from Steve's experiences as a defensive coordinator -- the psychological impact on the opposing team's defense:

As I've mentioned, I was a defensive coordinator.  When I saw the opposition bring out their punting unit, I felt a sense of victory.  We had a very good team , and our defense was considered dominant, so I felt this victory more often than not.  However, when an opposing offense kept the offense on the field on fourth down, I always felt a little concern.

I was confident in our players, and I wasn't "afraid."  But seeing a team stay on the field on fourth down makes several thoughts run through my mind.  "My guys can't rest yet.  The game isn't won yet."  As a DC, I  always felt relief to stop a team on third down.  When the punting unit doesn't come on the field, I get robbed of that good feeling.

So this is an argument purely from emotion.  In sum, "The DC of the opposition doesn't want you to go for it on fourth down.  For that reason, there must be merit in going for it on fourth down."

I know that sounds like sarcasm, but I mean it very seriously.  When I don't see the other team's punter take the field for fourth down, my first thought is, "Aw $h!t!"

Going for It on 4th Down: Bringing It All Together

    This has proven to be an intriguing discussion.  TJ's analysis of the New England choice to attempt a 4th down conversion deep in their own territory, late in a game, along with the case of the Pulaski Academy Bruins makes a strong case for a team going for it on 4th down.  In fact, in the three years that Coach Kelley has been using his philosophy, the Bruins have posted records of 9-2-1, 13-1, and 9-4.  They have made the playoffs each year, and won the state championship for their division in 2008.

    On the other hand, as Steve and TJ pointed out, there is a strong reticence among coaches to stick with the "tried and true," time-proven strategies and tactics that they have learned from other coaches.  As was mentioned, coaches are paid to win football games.   To win games, their offense must put points on the board, and their defense must keep the other team from scoring.  It is logical that when the offense is facing a 4th down, and are in field goal range, that the coach would opt for the field goal.  It also makes sense that when the offense is facing a fourth down deep in their own territory, that the coach would choose to punt the ball to force the opposing team to travel further to put up any points.  The cynic might make the claim that the coaches are taking the safe route to protect their own job.  More likely, the coaches are simply doing what they have seen other coaches and teams do successfully -- they are following the formula put forth by common wisdom.  Thus, they choose to not follow what some statistics seem to support: that they could be more effective by going for it on fourth down more often.

    Which brings us to another point which bears being raised: the nature of innovation in the game of football.  It's been written in a number of places that the game of football is a continuously evolving game.  An offense will try something new, so defenses have to adapt in order to stop the offense, which forces the offense to find new ways to best the defense, and so on and so on, etc.  The resurgence of the wildcat formation following Miami's successful use of it early on in 2008 is a good example.  Miami experienced early success, then defenses caught up and the formation became less successful.  

    Which brings to the forefront of the discussion why there has not been more of a move to institute a practice of going for it more in the NFL, ala the Pulaski Bruins.  No team in the NFL attempted to convert more than 26 fourth downs.  The New York Jets had a 75% conversion rate on fourth down, yet attempted to do so only 20 times.  By comparison, the Jets punted 80 times.  If a team is converting 75% of their fourth downs, would it not make sense to try at least a few more?  Part of the issue may reside in the difference in the relative effectiveness between the high school setting and the NFL.  The average net punt in the NFL in 2009 ranged from 34.7 to 43.9 yards.  Coach Kelley's experience is that the average net yardage on a high school punt is 30 yards.  Another factor may come from the fact that while Pulaski converts an average of 50% of their 4th downs, the NFL average in 2009 was 46.9%.  Thus we see in the NFL, there is a lower effective conversion rate on 4th downs, combined with a longer net punt average.  What works so well for Coach Kelley at the high school level may be much less effective at the NFL level.

    One last thought about this discussion:  while it makes logical sense that NFL coaches would be reluctant to make a standard practice out of attempting to convert every fourth down -- despite the fact that a preliminary look at the idea from an EPV point of view makes tremendous sense -- there is one reason that might provide a rationale for at least trying it more often than is currently being done:  the psychological impact on the opposing team.  To repeat a statement by Steve from his perspective as a defensive coordinator:

When I saw the opposition bring out their punting unit, I felt a sense of victory . . . However, when an opposing offense kept the offense on the field on fourth down, I always felt a little concern . . . seeing a team stay on the field on fourth down makes several thoughts run through my mind.  "My guys can't rest yet.  The game isn't won yet" . . . When I don't see the other team's punter take the field for fourth down, my first thought is, "Aw $h!t!"

Wouldn't it be fun to be able to make opposing Defensive Coordinators go "Aw $h!t" on a regular basis?

(photo courtesy of the Pulaski Academy website)

An Addition to the Article

     Today I received an email from none other than Coach Kelley of Pulaski Academy.  In that email, he commented on this post.  I received permission from Coach Kelley to share his comments with our readers here at MHR.  What follows is the text of the email he sent:

My name is Kevin Kelley and I am the head football coach at Pulaski Academy.  I saw your article yesterday and wanted to comment on it.  First, I am flattered that people continue to write about our methods.  I will admit, it is kind of fun although when we started doing it, we had absolutely no idea that media would like it.
 
Anyway, the numbers are there and they fit and point to the fact that people should use 4th down much, much more than they do now.  I have talked to college coaches, NFL guys, etc, as a byproduct of the media coverage it has received.  Several very prominent names agree but say that coaches will never do it because even though it might help them have a better percentage chance to win because their risk of losing their job overshadows that possible higher chance to win.  The irony is, they are obviously fired for losing.
 
The numbers do point to not punting nearly as much, but the reason I have forgone the punt almost totally is, like anything, when you experience it, you realize all the other things that go along with it that no one know about.
 
For instance, at any level, people spend a lot of time on special teams.  They say special teams are 1/3 of the game, blah, blah, blah.  If you look at the actual number of plays that are special teams in a game, you will see that it is simply not true.  Then if you take out extra points and defending extra points (they are rarely blocked, it is simply up to the kicker to miss it and he can kick while they are practicing), it really minimizes it.  Eliminate punting and you are really taking special teams down to a minimal level.
 
This gives you extra time to work on more offense and defensive strategy, situations, etc.  Another dynamic is the psychological factor.  We condition our kids in practice for that fourth down.  No one else does that like us because our kids know week after week that we will be going for it.  Put people in a situation of greater stress and their heart rate increases, breathing rate increases, mind races, and that simply gives us a bigger advantage on that one down as well as fatiguing the opposing team for later in the game.
Psychologically, it affects the other team as well.  Here is an example.  When a team acquires the ball via a turnover, they are almost twice as likely to score a touchdown.  I think this has to be a mental thing as there is no other explanation.  When we convert a single fourth down on a particular drive, we are score a TD on those drives twice as often as we scored on drives before we started doing this stuff.
 
Those are just some examples of the dynamics outside the numbers that not punting bring into play.  Just thought I would share some thoughts.
 
Thanks for your time,