Lol, true.
It's the guys that don't talk crap you need to watch out for. My first instructor I had was about 5' 10 170, average looking guy. This guy could still kick the crap out of me. I train with him about once a week, and his skill is seriously world class. If he wanted to compete in the UFC, he could. I'm bigger and stronger and fairly skilled, he even told me that my brother and I are his best students. Every time we spar or grapple, he always wins. I can handle him in grappling because of the strength advantage, but if anyone ever taps, it's me. We usually go over 5 minutes without a tap.
Sparring he is virtually unbeatable. His timing, accuracy and dexterity are impressive to say the least. He has 3 National titles in Karate competition. Super quick, and he kicks like a horse. I love sparring him because it's a challenge, and I'm humbled every time.
Anyways, he's the type of guy that's well dressed, average guy, really nice, doesn't strut, but he breathes martial arts and he would absolutely murder anyone that tries to mess with him.
There are lots of people out there that are dangerous that don't compete in MMA. MMA definitely attracts hardasses but there are, like you said, lots of true hardcore guys that don't act like tough shit.
Lyoto Machida is a prime example of this type of guy.