for sometime about how to introduce this to the crew here. I finally settled on just throwing it out there. It will make more sense as time goes by. Some pointers:

Right now, the only relevant score is the Overall Score. Everything else is simply to small of a sample size to draw conclusions from yet. Overall score isn’t great yet either, but with two games it will be much better, and the goal is to have it pinpoint at the end of the regular season.

Don’t get caught up on where the numbers are coming from. For now you will have to trust that they are consistent, but I welcome everyone to challenge whether or not they are RIGHT. That is how the system will get more accurate.

When reading the numbers, the Overall Score is comparable from one player to the next. Obviously you can’t compare a runblock score to a short pass score, and you aren’t supposed to. The only score that is universal is the overall. The other scores are only listed to help understand and compare like positions, i.e. one QB to another.

The numbers will really start to make sense when you see them changing, from play to play and game to game. I feel like I have more of a grasp of it because I compiled these from the game log, and so I saw them changing as each individual play was added. I would have posted that, but I don’t have that kind of time. I mean, we are talking hours and hours.

I hope the grades will make more sense as I field questions about it. In that light, I’m off to answer bfsisd’s question below…

just deleted my whole answer. Here are some points to keep in mind:

Overall score is the most important number. That number allows Cutler to be compared not only to Hackney but to Kuper as well. That is the whole point of this, and please keep in mind this is a fledgeling system. It has a loooong ways to go.

Don’t get caught up in where the numbers are coming from. They aren’t based on stats, they are based on a numerical model that represents the game. Because the model is built play by play, and player by player it takes into account the context needed to judge the plays. Nothing in a vaccuum could be the NPLB motto.

One major error I made was not explaining up front how to read the chart itself. Obviously overall score is easy. Short medium and long are arbitrary breakdowns of the individual plays, meant to give a look at a QBs development in regards to what part of the field (and thus what level of defense) they are proficient at reading. The number that follows is the number of plays that fit that category and the QBs score in that isolated area.

Any number of criteria could be selected, including which side of the field he is throwing to, which receivers he is targeting etc. Some of that info is included in the analysis that accompanies the chart, but I simply can’t include everything. There is too much. I selected depth of pass because I think a lot of info can be gleaned from it, but other criteria could be selected.

I also included throws under pressure for obvious reasons, and dropped balls for equally obvious reasons. It isn’t enough to know that someone went 22 of 31. Skill wise that player may be a better player than that suggests.

I have included checkdowns because I had some interesting data for it, but it isn’t a fundamental number or anything.

Just remember that the most important number is Overall Score. It is based on individual scouts and rankings of each player on each play they participated in. There are literally hundreds of other numbers that could be calculated from those scouted plays by being selective about which plays you look at. I can choose any of them, and by request I will. What i have chosen above are some areas that seem relevant to me.

First a clarification. When cutler is rated at +30 overall it is a grade, like an A, B or C, not a number of points. When I say that he is rated at -34 on checkdowns I am saying that on the plays where he checked down his technique, accuracy, decision making etc. grades out poorly.

But in this case the sample size is too small. Cutler had one average play and one poor play on checkdowns. If he had had one more average play his score would have been closer to -17. Add a great play in after that, such as escaping from pressure and making a great throw, and the score could jump significantly. So far I have found that the minimum number of plays needed to assess an overall score is around 20 for QBs. That isn’t a lot which makes it valuable for single games. The greater the number of variables, the more plays you need. WRs and RBs have fewer variables and require fewer plays. Checkdowns have a lot of variables, and I would estimate that you would need at least 5 plays in a short time period (one game) to get any accuracy.

From college scouting, this is fairly similar. To get a read on a DT for example, you usually needed about 5-6 rushing plays and 5-6 passing plays in a short period of time. More plays meant more accuracy. But after a certain point the accuracy is raised only by very small amounts. That threshold should be the minimum numbers of plays required and it will be a learning process to figure out which kind of plays can be analyzed like this.

Also, regarding the pressure, cutler and ramsey both made bad throws from a technical standpoint, while Hackney ran to the line of scrimmage and fell down in a relatively safe play. Hackney’s play will get a team nowhere, but Ram and Cut’s plays could severely hurt a team. Again, small sample size also applies. More pressures would have given a better reading on not only the act of throwing and decision making under pressure, but would also have created a new thread of inquiry involving adaptation to pressure.

is stat based, so it is fundamentally different, however the FO guys are diehards, and extremely smart, so what they see bleeds over into the data they compile. The result is Prospectus. It is a great work and I have a tremendous amount of respect for what they do, but I don’t agree with it.

As long as stats are part of the equation, there is an inbuilt % of error that they must accomodate, that increases with every complexity they attempt to describe. I’m not certain if it is a “chasing the carrot” situation or not, but it is pretty damn close.

This system, in theory, should not suffer from the degree of sample size issues that plague a stat based method, since the statbased method needs to have enough in the sample to overcome the inbuilt % of error inherent in the inaccuracy of stats.

Because it is based solely on scouting reports, what it describes is over 90% accurate, and the only additional error that is added occurs at the single calculation that that transforms the scouting report into a grade. If the grade was from 1-3, that would be a high degree of inaccuracy. If it was from 1-1,000,000, obviously it would be completely negligible inaccuracy.

That is the main point of debate among the analysts right now: objectively grading different flaws and virtues. The requirement is only that they be in the right order from important to non-important, and that it is consistent from one example to the next.

I too am excited to see where this ends up going, and I will pass your good wishes onto the analysts.

players on other teams will become available as the season goes on. Expect to see individual game comparisons of our opponents, but as far as tracking and compiling for other teams and players, that is too much work right now and will be planned into the future.

Head Analyst from the NPLB notes in a comment up above that they will try to keep track of some of the top QBs int he league this year.

Its not much, but hopefully it will keep you from storming the castle for now…

and claiming no (or less) error is tooting some serious horn.

But I stand by it to a degree. When you say that there is a large bias I agree completely. The bias of the analyst (the weighting if you will) is directly proportionate to the amount of detail he includes, with each detail being an individual observation that must be interpreted. This system contains a TON of bias.

But when you say that it therefore has a large % of error, I have to disagree. One doesn’t follow directly from the other. It is more accurate to say that it could have a large % of error. But what if the bias is a bias towards truth? That by its nature will lower the % of error.

On the other hand, stats always have a certain % of error. That is because they are not designed to measure the skill of the player, but instead are designed to concretize a particular event. They can only give you data about the event, so extrapolating that out to an assessment of the player means you need more data.

That is why FO is so complex. They are dealing with a system 32 times larger than mine, at the very least. To start with, they need a lot of data. I don’t know what they compile for themselves, but they certainly rely on the records of company’s like Stats Inc. Those companies are motivated by the need to generate lots of useful data on the games, and event data is simply the easiest and quickest to collect. (the data is getting better all the time, of course).

FO, in order to create statements about teams (Denver has a more effective rushing attack than 80% of the league) and in order to make statements about players (Marvin Harrison is the most effective receiver in the league), have to mix and match loads of different event data (the mixes of course are their custom metrics) such as Marvin Harrison caught a pass from Peyton Manning on 3rd and goal, one of 13 such passes he caught all season, in an attempt to craft a formula that translates one to the other.

They take an assessment of events and extrapolate out an assessment of players or teams. They do amazing work in that arena, with mind numbingly complex calculations. But to me they have inserted an unnecessary degree of inaccuracy into the process.

For someone who is scouting a player, the accuracy is completely dependent on whether they know what is truth or not. Stats aren’t concerned with truth, being literal recordings of events, so someone can use them to tell a lie. (More accurately someone can tell a lie by using them. No stat can truly lie.) FO relies on their OWN understanding of truth to evaluate what formulas do and don’t work. They refine according to that belief in what is true. Just like them an analyst using this system is dependent on his own belief in what is true. FO has a reputation for quality because their belief in what is true about football meshes well with the real thing, and they have a record of changing it if it doesn’t. The requirement of a scout grading a player is that he have a belief in what is true that meshes with what is really true about football.

Both systems have the same risk of error in the bias area. Stats additionally introduce an intermediate step between what is and what is known that compounds the error.

This system could be horribly skewed towards a very untrue and unprofessional vision of what pro-football is all about. The NPLB is relying on this community for that not to happen, and to keep its focus on what is truly the TRUTH.

:)

I agree that math tests and English tests offer differently levels of subjectivity vs. objectivity, but depending on the teacher/professor, even math can have its subjectivity.

In college I had some math professors that would give partial credit for answers as long as you got the basic problem correct but did something like miss a sign (which I did often) and others that looked at the answer and based the “correctness” based on whether is was right or not no matter what.

I guess that’s more of a case of leniency as opposed to subjectivity…