YMMV.
Look at american football players. I have played with guys that were absolute beasts who would have "case races" on weekends. Same goes for weed smokers.
I personally don't do either.
I guess someone has to do a study with a bunch of sets of identical twins. One out of each pair gets 'faced a few times a week, the other abstains. Otherwise same diet, training regimen, etc. See who has better size/strength/athletic ability at then end of a few months.