I agree with a lot of the comments.......
You haven't done anything wrong & you can't give this person the satisfaction of not going to the gym or not talking to people as that's just not YOU.....
Plus, there are a LOT of people in this world who enjoy, live for & get off talking badly about others, trying to cut others down, etc.....Don't let them get you down!!