Cespedes' single came one batter before Callaspo connected for his second home run of the season. John Jaso drew Cosart's fourth walk of the first inning before Reddick hit his first homer with his own towering drive to right field.
That chased Cosart, who retired only one batter before being replaced by Paul Clemens.
Gray (3-0) gave up more than one earned run for the first time this season. He allowed RBI singles to Dexter Fowler and George Springer in the fifth along with Chris Carter's sacrifice fly before Crisp homered in the bottom half.
Oakland won its ninth in 11 games after going 7-2 on the club's first road trip. The A's had their four-game winning streak snapped Wednesday at the Angels, but are the only team in the majors yet to lose back-to-back games.
Houston manager Bo Porter left the dugout in the third to yell at Jed Lowrie, apparently upset that the No. 2 hitter attempted to bunt with his team ahead 7-0 in his second plate appearance of the first inning.
Lowrie flied out in the third after two inside pitches.
Cosart received a mound visit in the first after consecutive one-out walks to Josh Donaldson and Brandon Moss loaded the bases. After a high pitch near Cespedes' head, the Cuban slugger lined a single into center field.
Cosart's quick exit ended a solid stretch in which Astros starters posted a 3.00 ERA over the previous seven games.
Gray allowed nine hits, struck out four and walked two in six innings as the A's handed Houston its fifth straight loss and sixth in seven games.
Springer went 2 for 5 while batting cleanup for the first time, two days after making his major league debut. Porter said he would have preferred to avoid putting the prized prospect in that position in his third game, but the manager wanted to shake up the lineup to get the Astros going.
Oakland's eight runs in the first two innings were more than the seven Houston totaled in its previous four.
NOTES: Reddick homered for the first time in 87 at-bats since Sept. 15. It was his fifth hit of the year. ... A's LHP Scott Kazmir (2-0) is set to start Saturday after experiencing arm trouble in his last outing. ... Springer got his first stolen base. ... The Astros went 3-7 at the Coliseum last season and 4-15 overall against the A's. ... Oakland pitcher A.J. Griffin, sidelined since the start of the season with an elbow injury, is yet to begin a throwing program and manager Bob Melvin isn't sure when that will happen. ... Gray is 5-1 in nine career games against AL West teams.