They have been outstanding ever since, and are still chasing their first NFL title since the 2004 season.
''I think when you're young, there is no perspective,'' the 37-year-old quarterback said Wednesday. ''Now that I have a little perspective on it, yeah, it's hard to do. To do what we accomplished in a short period of time was amazing.''
The Patriots have been to two Super Bowls in the last nine seasons. Helped by two late receptions - one on David Tyree's trap against his helmet, the other on a long throw down the sideline to Mario Manningham - the New York Giants won close games.
If those passes were incomplete, Brady might have a Super Bowl ring for each finger of one hand.
''We've come up short a few times in some big ones, but we're lucky to be in those games, too,'' he said after the Patriots held a rare scrimmage during practice to motivate them during their bye week.
''It was fun to have a real competitive day,'' Brady said. ''I could see why it's so hard to score on our defense, though, I can tell you that. They've got a lot of good players.''
In the blue vs. white scrimmage, the white team won, cornerback Brandon Browner said - his team.
''We won by a field goal,'' he said. ''It was a great way to keep guys competing and get something out of practice because it is a bye week so you've got some guys, their minds are on maybe getting out of here or trying to get done with the week.''
The Patriots (12-4), the AFC's top seed, practice Thursday, and then have three days off before a weightlifting day Monday. They will start practicing on Tuesday for their divisional game.
Many of them will watch this weekend's playoff games that will determine their opponent on Jan. 10 in Gillette Stadium.
Whoever it is, the Patriots offense needs to improve over its last two performances, the least productive of the season.
New England gained just 231 yards but edged the weak New York Jets 17-16 on Dec. 21. Then they picked up 260 in a 17-9 loss to Buffalo, although Brady played just the first half and tight end Rob Gronkowski and wide receiver Julian Edelman didn't play at all because the game had no bearing on their postseason seeding.
And the Patriots have beaten three other AFC playoff teams - Cincinnati, Denver and Indianapolis - by at least 22 points.
''You just have to believe in what you're doing and believe in each other and go out there and play with a lot of confidence,'' Brady said. ''In spurts we've done some really good things, and other times our execution hasn't been as sharp.
''It really doesn't matter at this point what has happened - good, bad or indifferent. It just really matters what we really do from this point on. Every team has the same record.''
Brady won his first 10 playoff games. But in the last five seasons, he is 4-5 with losses in the Patriots' first game in the first two years, and then the Super Bowl loss followed by losses in the AFC championship game the past two years.
''To be that final team is pretty tough,'' he said, ''but we've also put ourselves in a good position, so that's all you can ask for at this point for us.
''We look ahead to it with confidence knowing that we are capable of playing the best teams and playing really well against some of the best teams. All year we've done that, and now we've just got to do it in a single-elimination tournament.''
Online: AP NFL website: www.pro32.ap.org and www.twitter.com/AP-NFL