The Panthers (3-1) nearly squandered the 12-point lead they had at the 5-minute mark when Torrion Brummitt's layup pulled the Catamounts (1-2) within 69-65 with 1:05 left.
After Brown's free throws pushed the lead back to six, Haiishen McIntyre made all four of his foul-line attempts in the final 21 seconds.
Lorenzo Cugini added 13 points and Austin White scored 12 for High Point, which has won 21 of its last 23 home games. Brown drew cheers from the crowd with a dunk midway through the second half when he finished McIntyre's alley oop assist with a high-flying finish off a baseline cut.
Brummitt led Western Carolina with 18 points. Mike Brown scored 14, Haboubacar Mutombo 12 and Rhett Harrelson 11.