Yes, the "it" factor is a big part of it. And being single. There has to be a connection, chemistry, and for me I wouldn't sleep with a chick just because she was hot. This is one reason why I don't like alcohol to be a deciding force on this issue, because impaired judgement can spell disaster. There are times when I could be really horny and if I see some chick at a mall or something that's just too hot, and she let me have her right then and there, I'd be tempted as hell to bone as long as I had a rubber. Never had that happen, but those would be the toughest times to uphold ones character, at least for me.
But then my conscience kicks in and I start thinking about afterward and how it would just be a memory.
If I had a one night stand of all out sex, it would ideally be with a girl I hit it off with and was in a situation(day or night) that was too good to pass up, and would probably never get the chance to do it again.