Interesting logic
I'll indulge your idea for a moment here though. Let's roll with the genetic thing. If it is genetic then they have no choice in who they find attractive. They could choose to ignore those feelings but it would probably be just as easy as you ignoring your desires for women and sleeping with a man all the time.