The Computer Poll Uprising: Creators of the BCS's Most Controversial Component Look Back
When Richard Billingsley began ranking college football teams in the 1960s, he needed just a few things: a newspaper, a pen, paper, and his Olivetti hand-crank adding machine. It would be nearly 30 years before he had access to a computer. Thus for three decades, the future BCS computer pollster was something more like a notepad pollster. It doesn’t have the same ring.
In those days, Billingsley was merely a rabid college football fan in Houston whose rankings were good enough to be published in a neighborhood newspaper. Every Sunday, at about 8 a.m., he’d begin his work, which often took him all day. Sometimes, he had to wait until the Monday morning edition of the Houston Chronicle to collect the last of his scores—especially if Hawaii had played a late-night game. Even one missing score would completely skew any set of ratings.
Billingsley was one of the earliest sports rankings hobbyists. Another, Richard Dunkel, began his own process in 1929 with a stack of index cards and had carved out a niche in the industry, as had Herman Matthews, who began rating college football teams in the late 1940s while in high school. Their rankings were enjoyed only by rabid college football fans who happened to subscribe to the newspapers in which the hand-calculated data was published.
Not long after Matthews began ranking teams, an elementary-school student named Jeff Sagarin got into the game. When Pro Football Weekly bought the right to publish Sagarin’s ratings in 1972 on the path to the mainstream, a trail had been blazed for many of those who’d follow him into this budding industry. Sagarin had just graduated from MIT, and he loved sports. Crucially, he also had access to a computer.
“Back then, there was no email,” Sagarin says. “In fact, in the winter of ’84, I was still using a mainframe computer. I didn’t even have a PC. I got my first PC in September of ’84. I would print them out on the mainframe computer at Indiana University, and then I would run to the post office and send them … by express mail. I don’t think there was email that I knew about then, maybe CompuServe—so maybe, I wonder if they had a modem I could send it to by then?”
In 1984, the NCAA tournament committee first used Sagarin’s ratings to help determine the field of 64. A year later, USA Today began to publish the ratings, which afforded him a more forward-facing persona than his peers and in turn inspired a second generation of pollsters, who emerged from a demographic that could conservatively be characterized as “nerds”, a population which had only rarely made its presence felt in the larger college football conversation. Starting in 1998, several of those pollsters spent more than a decade at the center of the college football universe, the men behind the computers that accounted for one-third of the BCS rankings.
Over the BCS’s 16 years of existence, 10 different computer ranking systems had a say in the two teams that got to square off for college football’s biggest game. Four of those were only briefly incorporated: the New York Times rankings (1998–2000), Dunkel’s DKC Ratings (1999–2000), the Herman Matthews/Scripps Howard rankings (1999–2001), and David Rothman’s FACT College Football Standings (1999–2001). As their ranks shifted, so did attitudes toward them and the rules they faced.
This is the story of the six computer polls—and the people behind them—who swayed the BCS countless times, for better and worse, and saw their rankings through to the bitter end.
Andy Lyons/Getty Images
Jeff Sagarin: The professional
BCS pollster from 1998 to 2013
2018: Still earns his income through publication of his rankings
Sagarin’s rankings were one of the three original computer rankings used during the BCS’s inaugural 1998 season, along with the Seattle Times ratings—the brainchild of Jeff Anderson and Chris Hester—and the New York Times ratings. The original three likely earned their spots because they were published in major media outlets; the BCS could hang its hat on the credibility those newspapers lent the ratings. When Roy Kramer, then the SEC commissioner, began to construct how the BCS would determine its rankings, he called Sagarin and asked if he’d like to be included.
Sagarin: “[Kramer] said, ‘We’re going to have two other ratings.’ I said, ‘Well how come you’re using these other two ratings? Why can’t you just use mine?’ He said something like, ‘Jeff, you’re the only one who would think that’s a good idea.’”
Anderson: “[Kramer] was very much a believer that there had to be some objective quality. You couldn’t just rely on the polls and their subjectivity. He didn’t want a committee and its subjectivity.”
Sagarin: “That first year was real exciting. Let’s just say, I shot my mouth off a lot. People in the press loved me. You guys called me almost every week. They knew I would say something incendiary. Seemingly every week, I’d get a phone call from Roy Kramer. The first 30 seconds, he’d be cursing at me. Then he’d start laughing, and we’d talk for 20 minutes.”
Jeff Anderson: The upstart
BCS pollster from 1998 to 2013
2018: Director of the Bureau of Justice Statistics at the Department of Justice
Anderson and Chris Hester had first gotten their ratings published by crticizing another original BCS ranking system. In the early 1990s, Anderson had been inspired by how bad he thought the New York Times rankings were, and he and Hester spent a year developing their own alternative. They sold their finished product to the Seattle Times in 1994.
Anderson: “[We said to the Seattle Times,] ‘You’re running the New York Times computer rankings. If we could convince you that ours is more accurate, would you consider making a switch?’ We put a presentation together ... and they agreed to run our rankings. They called them the Seattle Times Rankings and agreed to pay us the princely sum of $25 per week. We would have gladly paid them, so we were ecstatic.”
The computer polls were there to take the BCS rankings beyond just records, to help people see behind an undefeated team with a weak schedule and a two-loss team that had played a gauntlet. The inaugural season of the new system was a wild one, and its final week provided the first major test.
With three teams still undefeated on the morning of Dec. 5, the BCS had Tennessee (11–0) ranked No. 1, Kansas State (10–0) No. 2 and UCLA (11–0) No. 3. At No. 4 was 11–1 Florida State—but that seemed almost irrelevant.
Anderson: “I still remember the crazy last day of that season. It was one of the nuttiest ones ever.”
Sagarin: “People were in a frenzy. Oh my god, if all three of them win this Saturday, what’s going to happen?”
Of course, Kansas State blew a 15-point lead to Texas A&M and lost, and UCLA lost to Miami in a game that had been postponed by a hurricane. That night, Tennessee fell behind Mississippi State in the fourth quarter on an 83-yard punt return touchdown—but the Volunteers struck back on their ensuing possession to retake the lead, pulled away for a 24–14 win and locked up a spot in the title game. The identity of No. 2, on the other hand, was far from certain.
Anderson: “Next thing you know, I was getting calls, I think from ABC. They had this selection show, and they were asking what’s going to happen. They had no idea. I said the only thing for certain at this point is Tennessee’s going to be in, and Florida State ended up making it. I faxed those rankings off at 2 a.m. ET to the SEC offices, got a confirmation they received them, and that was the end of a pretty incredible day and season.”
Sagarin: “I’m still horrified by [Kansas State’s] loss. It would only happen because I was riding on their back. … I stuck to my numbers and still had Kansas State No. 1 when the dust settled. … To be about to go to the championship and then get sent to the Alamo Bowl, wow.
“I was getting unbelievable emails from people. Unbelievable. This wasn’t media, this was just people, because my email address is published on my ratings. Sometimes initially I would try to respond, because I thought my incredible reasoning power would make people say, ‘Oh, you’re right Jeff. I hadn’t looked at it that way. My apologies for making such a dumb mistake.’ I never received an email back saying that. It was always piling on. It occurred to me that people were not always impressed by my reasoning power.”
Courtesy of Richard Billingsley
Richard Billingsley: The historian
BCS pollster from 1999 to 2013
2018: Medical biller
Billingsley is the exception to the rule among the pollsters: he wasn’t a career mathematician or statistician. Instead, he spent much of his life as a minister. College football, though, was always a passion, which is how he came to apply his rating system to all of the sport’s history dating back to 1863. Billingsley ran every single team through his formula, and when the NCAA caught wind, it was impressed, and it published his work.
Kenneth Massey: The professor
BCS pollster from 1999 to 2013
2018: Professor of mathematics at Carson-Newman University
Massey developed his ranking method that eventually helped pick college football’s national champion while working on his undergraduate thesis in the early ’90s, and he still uses it—but now, he applies it to the Carson-Newman Eagles.
“One of the reasons I like what I do is I can rank, let’s say, Division-III volleyball teams without even seeing a game or even really knowing strategies or anything about the sport,” Massey says. “It’s such a general algorithm, it works for any sport you want to throw at it.”
At Carson-Newman, the proprietor of the Massey Ratings has been known to follow the Eagles’ softball team. In the South Atlantic Conference where it plays, reputable ranking information is hard to come by at times, and Massey often wonders if his school’s teams are getting proper credit. “They’re kind of regional, those lower divisions, and it’s hard to know where you stand relative to a national ranking,” he says. “I think it’s interesting to be able to apply the same model and know whether our soccer team is likely to win this weekend.”
Peter Wolfe: The surgeon
BCS pollster from 2001 to 2013
2018: Infection disease specialist in Los Angeles
Wolfe, whose rating system employed a probability device called a maximum likelihood estimate, was always the most private of the computer pollsters. Although he did respond to a request to talk about his experiences for this story, he never replied to an emailed list of questions.
Wes Colley: The football player’s brother
BCS pollster from 2001 to 2013
2018: Holder of a Ph.D. in astrophysical sciences, now a member of the technical staff at Torch Technologies in Alabama, modeling and simulating missile defense programs
Colley developed the idea for his ratings formula while talking with his thesis advisor, who as a hobby came up with automated systems to rank things that do not empirically seem rankable. “Well, you take the length of the Encyclopedia Brittanica article about the person, but then you’ve got to go to the Chinese encyclopedia, because they’re important too,” Colley says in explanation of his mentor’s methos. “And it’s just, he has all these ideas about, if you look up the Wikipedia article about somebody, and you divide [its word count] by how long it’s been since they died. ... You know, Caesar’s not as important as Britney Spears. So he’s talking about all this stuff, and I’m sitting there thinking, Surely college football is easier than all that.”
When the BCS came calling, Kramer initially reached out to Colley’s father, who thought the SEC commissioner must be calling for his other son Will, a former Georgia tight end. “But finally he pieced together that it’s the other son,” Colley recalls. “Why does the SEC commissioner want to talk to my nerdy son?”
In 2001, the Dunkel Index and the New York Times ratings were dropped from the BCS formula, and Herman Matthews’s and David Rothman’s ratings left the field the next year. Thus, entering the 2002 season, the field of computer polls that would stay in the fold for the BCS’s duration was set, and the men in charge of those polls had fully adapted to the modern world, in which technology was king and college football was popular enough to afford any and all fringe players in the BCS equation some level of fame.
Massey: “There was actually more attention in the early years because it was new. At that point I was in my early 20s, and I’d never done an interview before. People wanted to write articles and even put me on TV a couple of times, and I hated that. I was awkward.”
Colley: “I just hung my website off of the astronomy website at Princeton originally, and then off the astronomy website at Harvard when I went up to do my post-doc. I actually got in trouble at Harvard because my sports website was getting more hits than any other site at the Center for Astrophysics, and they thought this was disgraceful. I thought, I’d better buy a dot-com.”
Massey: “I knew basically nothing [about building a website]. I like to brag to people that one of the models that I researched for my sports ratings was this thing called Markov chains, which was the exact same model that Larry Page used to start Google. Unfortunately, instead of deciding to apply it to search engines, I decided to apply it to college football. That’s why I’m not a billionaire today.”
The men were awkward, sure, but they were also outspoken, especially when the BCS changed the rules it used to govern the computers and the rankings in general. In 2002, it decided to drop margin of victory from the computer polls after Washington and Oregon had seen 11–1 campaigns marked by close, hard-fought wins come up short in the eyes of the rankings in consecutive seasons. Mike Tranghese, at the time the commissioner of the Big East, summed up the move this way: “A computer can’t get at the nuances of a score.”
Sagarin: “Somebody should have raised their hand and said, ‘Well just what are the nuances of a 59–3 score?’”
Massey: “It never was a big component, but they were just paranoid, so we had to take margin of victory completely out of the equation.”
Sagarin: “I wasn’t going to bastardize my ratings, so I just invented another column [excluding margin of victory] for the BCS.
“There’s no debate with [the BCS]. They would tell you, either you do this, or your gone. [Matthews and Rothman’s] systems had a lot in [them] that didn’t let runaway scores totally dominate. I respected them for quitting.”
Billingsley: “I still run [a version of my formula] with margin of victory, and it’s slightly more accurate in terms of future predictions of games, to the tune of 2 to 3%—which doesn’t sound like a lot, but really when you get down to looking at the games, 2% of difference is quite a bit. It’s a question of do you want a national championship that’s the best team or the most deserving team?”
Anderson: “Margin of victory is not the object of the game, and if it’s going to be incorporated in a ranking—and we didn’t really think it should be—it should be in the polls. They can tell the difference between a 28–7 game where a team gets a couple of late scores and makes it look close and the opposite.”
Sagarin: “When people would talk about the computers, they would derisively say, Oh, the computers are going to love this, this team is running up the score tonight. That would be a derisive phrase. Oh how shameful, that the computers would use scores.”
Anderson: “It’s always been a crazy thing to me; college football is the only sport where anybody cares about margin of victory. What was Golden State’s margin of victory over Houston in the Western Conference Finals? It was just phenomenal. They were probably 70 points ahead of the Rockets before Game 7, but if the Rockets won Game 7, that would have been it. The Mazeroski World Series in 1960, the Yankees [scored almost twice as many runs as] the Pirates.”
Then, in 2005, another tweak. In 2003, the BCS had selected LSU and Oklahoma for its title game, but the AP poll anointed USC the national champion; the next year undefeated Auburn was left out of the top two, while Texas lobbied the human polls and worked its way to a No. 4 ranking to secure a Rose Bowl bid over a deserving Cal team. From then on the rankings became an average of three calculations: the Harris Interactive Poll, brought in to replace the AP; the Coaches’ Poll and the computer rankings. The fact that subjective human rankings accounted for two-thirds of the system was a slap in the face to some—but not everyone—on the computer side.
Anderson: “Every year [before 2004], they [had been] doing kind of piecemeal tweaks. They’d make the change to solve last year’s problem, and they weren’t always well-conceived, and sometimes they came back to haunt the system.
Sagarin: “There’s no ratings that fit all the games. People get upset, because they pick out what’s important to them. They say, backfit it with these games. And you even try to say, but what about these games? They don’t even matter.”
Anderson: “That new formula was in effect the last 10 years of the BCS, 10 out of the 16 years. And in those last 10 years, there was never really any controversy about who was in the [title] game. That’s remarkable, to go an entire decade and never have a year where there wasn’t a consensus pick. Even in years like 2004, when Auburn was the only major undefeated team to ever be left out, everybody agreed with that: the polls, every computer, everybody.”
Colley: “I think people paid more attention to the computers in the BCS era, so they had a little more influence than just the one-third, in my opinion. The people who are polled for their opinion have listened to computer rankings that have informed that opinion. I think it’s a shame that’s gone away.”
Billingsley: “I don’t feel like the Bowl Championship Series ever did a great job of marketing how the computers played a role in the formula. I just don’t feel like the public had an opportunity to really have a very good understanding of the role we play.”
Sagarin: “What they were trying to find out was how much weight could they give to the computers. What was the maximum weight they could give to the computers whereby we really wouldn’t effect the results? We were there to look like guys in white lab coats, to give an intellectual aura. You could sense that we weren’t that popular.”
Billingsley: “I think the relationship between the computers and the Bowl Championship Series was deteriorating by the end of our working arrangement.”
The process became easier as the years passed: Gather scores, input into system, email rankings. It was a Sunday routine that was close to perfected.
The computers reclaimed center stage in 2008, when Massey’s ratings had undefeated Mountain West champ Utah ranked ahead of Alabama, which had been No. 1 in the human polls. At the time, it seemed ridiculous—even Massey agrees that it was borderline absurd. But, he says, once margin of victory went by the wayside, sometimes his poll results looked wonky. Alabama ended up facing Utah in the Sugar Bowl to end that season after losing to Florida in the SEC title game, and the Utes won handily, 31–17.
Two years later, in the final BCS rankings of 2010, LSU was incorrectly ranked ahead of Boise State, at No. 10 instead of No. 11. The issue was a result of Colley failing to input the Appalachian State–Western Illinois FCS playoff game that week—a small mistake, but one that still effected an order that helped determine bowl pairings. Within a community whose members adhered to varying levels of secrecy about the formulas that went into their rankings, Colley was the most transparent, and his error sparked a debate about how up-front the men behind the computers should be.
Some fans praised Colley in the wake of the error; others held him to a more exacting standard. Massey, though not as transparent as Colley, was always happy to share the basic tenets of his system, and he says the biggest distinctions between the models come in the “different tweaks and secret ingredients” that pollsters can layer onto their basic formulae; there just aren’t that many ways to go about ranking teams.
Jamie Squire/Getty Images
In 2012, the beleaguered BCS had been pushed to the brink of extinction by a controversial LSU-Alabama rematch in the previous title game. On June 20 of that year, conference commissioners met in Chicago and recommended a four-team playoff to take effect in 2014. Six days later, the College Football Playoff was approved, and the BCS’s days were officially numbered, leaving the computer pollsters to reflect on their run.
Anderson: “I tried to never assume we’d have another year. And I’m kind of floored that it lasted for 16. I thought along the way, the BCS actually became more popular ... but there was kind of a constant drumbeat for a playoff, which has gone back decades.”
Massey: “I always assumed it was inevitable. I think probably a four-team playoff is a good idea, but I always told people, the BCS was a playoff. It was a two-team playoff.”
Colley: “I think they just sort of kicked the baby out with the bathwater. The rankings had stabilized, and people were basically happy with the rankings. What they weren’t happy with was the one-game format. I applauded the move to the final four system, but I was like, why are you kicking out the rankings? That’s basically how I still feel.”
Massey: “I advocated pretty early on for a four-team playoff—just let the BCS formula select them, rather than go through this whole process of having an all-star committee and flying them in and wining and dining them.”
Colley: “When the BCS was running, the people were closer to the computers. Since the computers have gone, the people have sort of diverged again from the computers. I will say that one thing that started me into this game initially was I thought people were too easily enamored with name teams.”
Massey: “The BCS was great because it was completely automated. It was like a lightning rod, and if anything was unpopular, Condoleezza Rice didn’t have to get on TV and explain it.”
Sagarin: “Here’s the nirvana I’ve reached: I don’t care. It has no effect on my life if Oklahoma gets picked over Auburn or if out of the blue they decided that undefeated Arizona State gets in. There was a point in my life when I actually cared. I don’t care.”
Billingsley: “Today we have what most people a bonafide national championship game with our playoff formula, and we would have never had that if it hadn’t been for the BCS.”
Massey: “I’m sure people on the committee and people that vote in the polls and people that talk about college football on TV, I’m sure they’re still going to my website to check out what the computers are saying. It’s just another data point. I just enjoy doing it. Whether it’s part of the BCS or not, I’m going to keep doing it. Some people do woodworking, and some people go hunting, or whatever. I sit at my computer and do this kind of stuff.”
Billingsley: “I always knew that I would continue on with my rankings. I told the other computer guys, and I told [CFP executive director] Bill Hancock: My rankings were around a long time before the BCS came along, and they’re going to be around until the day I die.”
