Lafayette had lost four of its last five games and seven of its last nine before the win.
Rufer hit 5 of his 9 shots and 4 of 8 from 3-point range. Nick Lindner scored 13 points, and Paulius Zalys added 11 for the Leopards (5-11, 2-3 Patriot League).
Lafayette shot 45.8 percent overall and hit 10 3-pointers while holding the Crusaders (7-9, 2-3) to 32.1 percent shooting and 5 of 27 from deep.
The Leopards led 31-26 at halftime and extended their lead to double-figure (41-29) on a Linder trey with 14:29 left. A Robert Champion layup for Holy Cross cut its deficit to 55-48 with 6:39 to play, but the Crusaders got no closer.
Champion scored 24 to lead Holy Cross, which has lost three of its last four.