I don't blame you from considering going vegetarian. For many it is ideal. If you are very committed you will find that it is possible to get all your macronutrients while staying vegetarian. It is even possible to do it vegan, but I know that is not your thing.
I'm not vegetarian, but I am very sensitive to dairy meaning that I can't use the traditional supplements such as whey or casein protein. However, I haven't let this get in my way.
With proper education and discipline it is certainly possible to successfully build muscle while adhering to dietary restrictions.