The announcement comes less than a day after the Crusaders were ousted in the opening round of the Patriot League tournament by Bucknell.
Holy Cross finished this season 14-16, with an 8-10 record in conference play.
"I am disappointed that we must make this change, but the success of our men's basketball program has not been commensurate with our competitive expectations under Coach Brown's leadership," Holy Cross athletic director Nathan Pine said in a news release. "I thank Coach Brown for all his efforts at Holy Cross and his many contributions to the College community, especially his care and concern for our student-athletes."
In five seasons under Brown the Crusaders recorded a 69-83 record, with a 40-38 mark in conference play and zero NCAA Tournament appearances.
Pine added: "A personnel decision like this is always difficult and affects a great many people. We are fortunate to have very talented student-athletes who are committed to working hard in our men's basketball program, and I am optimistic for our future."
Prior to taking over at Holy Cross, Brown coached Mount St. Mary's for seven seasons, reaching the NCAA tournament in 2008.