Ely was 16 of 22 for 164 yards for the Rockets (4-0, 1-0). Terry Swanson rushed for 139 yards and Damion Jones-Martin finished with 85 yards and a score.
A pair of Ely touchdown passes in the second quarter, along with a Jones-Moore scoring run and a Jameson Vest field goal gave Toledo a 24-3 advantage at halftime.
Ball State's Morgan Hagee got the Cardinals (2-3, 1-1) on the board just before the break with a 29-yard field goal and Riley Neal hit Jordan Williams for a touchdown early in the fourth quarter to close the gap to 24-10.
With 5:34 left, Ely chewed up the clock with an 11-play, 57-yard drive to preserve the Toledo lead.
Neal had 236 yards and a touchdown for the Cardinals.