Delaware (6-4, 4-2 Colonial Athletic Association) took a 7-0 lead early in the second when Randolph scored from 1.
Will Fiacchi, who passed for 335 yards and four touchdowns, connected with Cole King and Nic Ketter to give Albany (6-4, 2-4) 14-7 advantage at the half, but Delaware used scoring runs by Eric Enderson and Wes Hill and a touchdown pass from Trent Hurley to Jerel Harrison to outscore the Great Danes 21-0 in the third quarter.
Fiacchi hit Josh Gontarek for a 29-yard touchdown on the first play of the fourth quarter to make it 28-21.
After the teams exchanged punts, Greenway hit the field goal to seal it.
Fiacchi added a 19-yard touchdown pass to Gontarek.