I agree. Whenever I'm able to plan my breaks, I always come back stronger. But that's as long as it's not more than a week off, after 3-4 months of solid training.
I'd say if you're having to take off more time than you want from the gym, go out and lift big rocks if you have to. Use whatever's available. It's not the same as your regular workout, but it's gotta be better than nothing.