Lee completed 24 of 35 passes for 275 and added 72 yards rushing, but threw five interceptions and lost a fumble. He threw seven interceptions last season.
The teams went into halftime tied at 14, but James Madison scored touchdowns on each of its first four possessions of the second half to take a 42-21 lead early in the fourth quarter.
James Madison, which finished with 625 total yards, has topped the 600-yard plateau each game this season.
Elliot Croskey had 102 yards rushing and a touchdown for Albany (1-2, 0-1).