The loss snaps the Dukes school-record 14 game regular season winning streak.
A Bryan Schor 7-yard TD pass to Rashard Davis cut the Dukes' (7-1, 4-1) deficit to 52-49 with 4:02 left. But Richmond responded on the ensuing possession, driving 69 yards on four plays capped by a Green 1-yard scoring run, pushing the lead back to 10.
A James Madison fumble by Cardon Johnson on the proceeding kickoff sealed the game for Richmond.
Vad Lee threw for 294 yards, ran for 124, and had five touchdowns for James Madison.
The teams combined for 1,305 yards of offense.