Nick Lindner made two free throws to pull Lafayette (2-7) within three with 1:16 left in overtime, but Townes made 1 of 2 from the stripe and, after an offensive foul on Lindner, a dunk by Potts made it 88-82.
Bryce Scott made a layup and, following a Fairleigh Dickinson turnover, hit a 3 to cut Lafayette's deficit to one with 25 seconds to play.
Jiggets' two free throws made it 90-87 a second later.
Matt Klinewski's putback for Lafayette off a missed 3-pointer made it a one-point game, but Anderson made 1 of 2 free throws and Klinewski missed a potential winning 3 in the final seconds.
Klinewski finished with 15 points and a career-high 16 rebounds.