Victor Oladipo scored 21 points and three others finished in double figures for the Magic, who shot just 35.6 percent from the field and made 3 of 11 3-pointers.
Despite that, Orlando managed to cut a 21-point Charlotte lead to 76-69 on Oladipo's 3-pointer with 8:58 left in the fourth quarter. But Neal's three-point play 26 seconds later pushed the Bobcats' lead back into double figures.
Already missing one starter with Josh McRoberts sitting out his second straight game with a sprained left ankle, Charlotte lost another late in the first quarter when Michael Kidd-Gilchrist went down hard on a blocked layup with 3:42 left.
Kidd-Gilchrist, who missed seven games earlier this season due to ankle problems, stayed curled up on the court for several minutes before rising unsteadily to his feet and walked to the locker room.
Team officials said that Kidd-Gilchrist had suffered a lower back bruise, but he returned to start the second half.
Jefferson had a double-double in the first half with 21 points and 11 rebounds, but it took a huge run in the second quarter to put the Bobcats in control with a 53-36 halftime lead.
Down 20-19 at the end of the first quarter, the Magic twice took brief leads in the opening minutes of the second quarter. They went up 21-20 on E'Twaun Moore's jumper with 11:43 left, and were ahead 25-24 after Oladipo's layup with 9:02 remaining.
But Tolliver's second 3-pointer of the quarter, coming with 8:44 left, put Charlotte ahead to stay and kicked off a 27-6 run. Jefferson's dunk with 3:36 left gave the Bobcats their first double-digit lead, at 41-30, and Walker hit a pair of free throws with 1:50 left to put Charlotte ahead 51-30, its biggest lead of the half.
Notes: Clifford said that McRoberts would take the next two days off, including Saturday's road game against the Cleveland Cavaliers, and may resume practice Monday. ... Magic C Nikola Vucevic was a late scratch from Friday's game for what team officials said was a sore Achilles tendon. His status for Saturday's home game against the Minnesota Timberwolves is unknown. ... The Bobcats have created a website - www.bigalspaint.com - to promote Jefferson as an All-NBA candidate. The website went public Thursday, and features quotes from opposing players and coaches, stats and stories by local and national media.