Head coach Doug Wojcik was fired earlier this month after an investigation into verbal and physical abuse of players.
GARDNER: Breaking down the 20 best big men
Young, who has been the Wofford's head coach for 12 seasons, cited his relationship with the team as his reason for not pursuing the Charleston job.
"The only thing that matters to me are those 16 guys coming to campus and this team," Young told The Spartanburg Herald-Journal. "I knew when it was time to back away and take care of my guys. I'm always for doing things the right way and winning another league championship."
Johnson, who played his college basketball for Charleston, said a domestic issue will prevent him from taking the job.
"I am ending my pursuit of the position of Head Men's Basketball Coach at the College of Charleston," Johnson, 39, said in an email. "I have a personal matter, my divorce in June of 2013, that requires my full attention, as my wife and I are working hard to reconcile."
- Paul Palladino