I've never withheld going down but I also never thought to "force it". I would find it odd, though if a man wasn't into it.
Never had a lover say, "No, that's ok sweetheart. Please dont do that." So I couldn't tell you what I would feel.
I get pleasure from giving pleasure so I have been told that my skill in this area (for those that I have wanted to give it to and that number is very low) is unparalelled.