Yeah, I'd say genetics are probably most important. Also, like you point out, some people claim to be natural when in fact they are only "clean" at a particular time.
I guess I'm personally just lucky in this regard as I'm hovering around 350lb at 6 feet 6 inches while still being fairly lean. I never had to resort to force feeding etc. in order to get to my current weight. Hell, I did not even use supplements. I was just born to be big, I think. It did take many years for me to go from 210lb to 350, though.
One thing that I had to learn to live with, however, is that it is actually not the right thing to do to claim to be natural, even if you are! I've experienced that when I compete in a powerlifting or strongman show, I'm always the biggest guy there, even when I know that 99% of the other guys are juicin'. When you then tell people that you are natural, they think that you are full of shit.
It's probably a case of guys thinking: "Yeah, right, I'm juicing and you are expecting me to believe that you are twice my size and natural!". The best thing to do is just not to bring up the subject at all, because what's the point if people are not going to believe you anyways. It's not that I've got anything against people who juice either. Being natural was a decision that I made and which I'm happy with, no matter if other people believe me or not.
Being natural, however, makes you LOOK different from a roid user, even if you weigh the same. It's sometimes subtle, but I'd say my built is more "normal" looking than dope users. I don't know how to explain this but I'm not as full of bulges and things as most of these guys. Then, of course, there are the dead giveaways that most roid users suffer from namely water retention, bad skin, balding etc. etc. which can be spotted from miles away.
Also, it looks pathetic when people like Ronnie Coleman make claims of being natural.