Dalembert opened overtime with a thunderous dunk and after fellow big man Rokas Gustys fouled out with 2:34 left, Dalembert made a layup for an 81-79 lead.
Ron Curry hit two free throws for a four-point lead with 25 seconds left, Hofstra did not hit the rim on a long 3-pointer and Dalembert sealed it from the line.
Dalembert also grabbed 16 rebounds for James Madison (14-5, 4-2 Colonial Athletic Association) and Shakir Brown had 22 points and 11 boards.
Curry hit 1 of 2 free throws with 23 seconds left in regulation for a 75-73 lead. Juan'ya Green's layup rimmed off, but Denton Koon's putback was good to tie it. Curry's 3-pointer was off the mark at the buzzer.
Gustys finished with 25 points and 20 rebounds - both career highs - for Hofstra (12-6, 4-2). It was the first 20/20 game for the Pride since David Taylor in the 1981-82 season.