Analyzing the predictive capability of recruiting rankings

It's easy to become disgruntled with recruiting rankings. Take Johnny Manziel, for example. Rivals.com rated Manziel as a three-star prospect in the class of 2011, and he went on to win the Heisman Trophy during his record-setting redshirt freshman season at Texas A&M. Rivals also listed Le'Veon Bell as a two-star recruit when he came out of Reynoldsburg, Ohio, in 2010. The Michigan State running back rushed for 1,793 yards in 2012 before declaring for the NFL draft as a junior.

The recruiting model misses on some players. That's inevitable given the differences in skill, maturity and level of competition among teenage boys. But with National Signing Day just two days away, a bigger question looms: How accurately do team recruiting rankings predict on-field performance? In the study below, I averaged and analyzed Rivals' team ratings from the previous four years to find out just how telling post-Signing Day rankings are -- and how they stack up to the preseason AP Poll as a predictive measure of success.

How to predict a college football season

To determine how recruiting rankings impact team performance, I ran a regression model on Rivals' recruiting ratings over a four-year period in order to predict each team's performance for the subsequent season. For example, a prediction for the 2012 season is based on recruiting ratings from 2009-12; a prediction for the 2009 season is based on the ratings from 2006-09. This regression technique provides a model that assigns a weight to each of the previous four seasons of recruiting rankings.

To measure team performance for a given season, I used the team rating from The Power Rank algorithm. This method considers margin of victory and accurately adjusts for strength of schedule. Teams are ranked based on a formula that outputs an expected margin of victory against an average FBS team. The team with the higher rating won 62.8 percent of bowl games over the last 11 years (219 out of 349), a higher mark than the 62.2 percent picked by the Vegas line.

Analysis was restricted to the top teams in the FBS, since the Rivals' ratings tend to differentiate between these schools to a much greater extent than they do the lower-ranked teams. For example, in 2012, top-ranked Alabama had 140 more points than second-ranked Texas. Yet the difference between the 84th-ranked team (FIU) and the 122nd-ranked team (Army) was just 141 points. This small margin between so many schools makes it difficult to accurately rank them. Teams from AQ conferences typically occupy the top of Rivals' rankings, but the ratings for Notre Dame, BYU and Boise State were also factored into this study, to better compare the Rivals' model with the preseason AP Poll.

Rivals' model performed as well or better than the preseason AP Poll on 46 of 100 teams

The regression model on recruiting data assigns each team a rating that outputs an expected margin of victory against an average team, as alluded to above. Sorting schools by this rating creates a ranking system that can be compared with the preseason AP Poll. To determine which rankings were more accurate for a given year, I assessed each ranking system against the final AP Poll. For each team in the preseason AP Top 25 from 2009-12, I determined which poll -- the regression Rivals' rating or the preseason AP Poll -- offered a closer approximation of the team's year-end ranking. For example, the Rivals' model ranked Alabama No. 1 in 2012, while the preseason AP Poll listed the Crimson Tide at No. 2. Since 'Bama won the national championship and topped the final AP Poll, the Rivals' model "won" for this particular team. In some circumstances, the two rankings can also tie.

By using the last four seasons in this study, there were 100 teams to analyze. The Rivals' model served as an equally good or more accurate predictor than the preseason AP Poll 46 out of the 100 times; this corresponds to a win-loss-tie record of 42-54-4. The comparison for the 2012 season is charted directly below this section.

Keep in mind: The Rivals' model only considers the talent level of players who signed a National Letter of Intent when they were in high school. The model does not account for coaching, on-field performance, early NFL draft declarations, injuries or any other extenuating circumstance. Considering that the preseason AP Poll does take these factors into consideration, the success of the Rivals' rankings is somewhat remarkable.

Rivals100: Class of 2013 recruiting rankings

A little bit of common sense makes the Rivals' model even more powerful. The site -- much like fellow recruiting sites Scout.com and 247Sports.com -- routinely underestimates the rankings of Boise State and TCU, smaller programs that have repeatedly busted the BCS over the last four years. During that span, Rivals never ranked Boise State higher than 61st or TCU higher than 34th. The preseason AP Poll has been more accurate seven out of eight times on these two schools, with TCU's 2012 season serving as the only exception. Excluding these two teams from the data set, the Rivals' model performed as well or better than the preseason AP Poll 45 of 92 times.

Moreover, the Rivals' model typically ranks Notre Dame very high. The Irish's No. 18 rank before the 2012 season was their lowest recruiting finish from 2009-12. The preseason AP Poll has been more accurate in predicting Notre Dame three of the last four years. The lone exception was this year: Brian Kelly's squad -- unranked in the preseason AP Poll -- went 12-0 before losing the BCS title game against Alabama.

Finally, it's worth noting that Rivals started publishing its team recruiting rankings in 2002 and has gotten more accurate as the years have passed. The Rivals' model for 2012 incorporated more data than the model for 2009; the Rivals' rankings went 8-16-1 against the preseason AP Poll in '09, but 16-9-0 in '12 (below). Rivals slightly changed how it calculates team ratings for the 2013 recruiting season. I expect this model to perform even better next season.

Four teams Rivals got right in 2012

First, let's look at two talent-rich teams that outperformed their preseason AP rank in 2012:

? Ohio State. Despite the Buckeyes' rich tradition and a new coach with the credentials of Urban Meyer, the preseason AP Poll had them listed at No. 18. However, Ohio State's success on the recruiting trail had it at No. 6 in the Rivals' model. The Buckeyes went undefeated and finished No. 3 in the final AP Top 25.

? Stanford. Given the departure of quarterback Andrew Luck and a pair of offensive linemen in the first two rounds of last year's NFL draft, the preseason AP Poll placed Stanford at No. 21. However, following recent recruiting developments, the Rivals' model ranked the Cardinal at No. 9. Stanford went on to win the Rose Bowl and finish the season No. 7 in the final AP Poll.

The Rivals' model can also identify several overrated teams that might not have the talent to support a lofty preseason ranking:

? West Virginia. The Mountaineers wrapped up the 2011 season with a resounding 70-33 Orange Bowl victory over Clemson, a performance that landed them at No. 11 in the 2012 preseason AP Poll. Yet after a hot start to the year, Dana Holgorsen's group sputtered through Big 12 conference play, eventually registering a 7-6 record. The Rivals' model had West Virginia at No. 39, which more accurately reflected a campaign in which West Virginia finished unranked in the final AP Poll.

? Michigan State. The preseason AP Poll had the Spartans at No. 13. But while their defense was elite again this season, the offense never played up to expectations despite the talents of Bell in the backfield. The Rivals' model ranked Michigan State at No. 32, and the 7-6 Spartans did not receive a single vote in the final AP Poll.

Why the 2013 recruiting rankings matter

The Rivals' model not only predicts future performance, it also provides insight into the dynamics of recruiting and team success. The regression model suggests that the most current recruiting rankings factor most heavily into a team's next-season results, which is slightly unexpected, as one might assume that ratings from two or three years ago should hold greater importance. After all, while freshmen continue to make bigger and bigger impacts, teams primarily rely on players with more experience.

However, teams coming off great seasons tend to land the best recruits. Consider Texas A&M this year, which has seen its pre-Signing Day recruiting ranking rise from No. 15 to No. 8 (as of Feb. 4) on the heels of its breakout 2012 campaign. The other top teams in Rivals' current rankings -- Florida, Alabama and Notre Dame -- all turned in terrific '12 seasons.

MANDEL: Under Sumlin, Texas A&M becoming a recruiting force

In sum: Recruiting rankings are more than a distraction used to pass the time until spring practice. Despite their extremely limited scope, they're nearly as accurate -- and in some cases, more so -- as the preseason AP Poll in predicting future on-field success.

For post-National Signing Day recruiting rankings analysis, check out ThePowerRank.com.