Cabrera, who is batting .308 with 14 home runs and 68 RBI, said he didn't want to mess with his swing by participating in the event.
"I'm going to enjoy watching," Cabrera said, via The Detroit News. "I've got enough issues."
Trout has 20 home runs and 63 RBIs this season for the Angels and leads the AL in OPS and is second in slugging percentage.
"I just want to be able to go and enjoy it," Trout said, via MLB.com. "I always had fun watching it, growing up as a kid watching it on TV, and being able to sit on the sidelines is pretty cool. I want to do it, and I don't want to do it."
The Home Run Derby will take place next Monday at Target Field in Minneapolis, the site of the All-Star Game.
Both players were selected as starters for the American League, and finished in the top two in the AL MVP race each of the last two seasons.