Baseball is a game of skill, not necessarily physical prowess.
You get top grade athletes in baseball for sure. Outfielders and middle infielders at the professional level have very impressive skill, speed and quickness. Just because they are buffoons in the gym doesn't mean they aren't athletic or strong or gifted.
I guarantee you if you stepped up to the plate against a 95mph you would shit your pants badly, and likewise if someone hit a 130mph line drive at your face from 90ft away. Pitchers have been killed from being hit from a comeback hit.
Lots of people think baseball is gay. It takes as much or more balls to play baseball than it does any other sport at the highest level perhaps besides motox/stunt sports.
The pros make it look casual because they are that good.
The hardest thing to do in any sport is hit a baseball. There's not another sport in the world that considers a 30% success rate excellent.
Also, the amount of steroid use in baseball is significantly less than it was 10 years ago. McGwire, Canseco, Sosa, Bagwell and lots of those guys were pretty jacked.