I know what you're saying.
"I eat 6 eggs/day, everyone else told me whole eggs are bad, mmkay?"
Thing is, my friend, the people they are advising don't work out!
If you are a healthy person, lift, do cardio, that really doesn't apply to you.
You aren't the average person, sadly.
I used to fret about eating 1 whole egg per day.
Went and got my physical, blood work, etc.
Healthy as a huge lean horse.
I eat at least 3 whole eggs/day now - still good. I don't think adding an additional 3 would not be a bad thing, just does not fit into my caloric intake/diet like it does with you.
If you are really that worried, to alleviate your mind, go get your blood and cholesterol checked.
Some people really have genetic issues with it, but more often then not, it causes an issue because they are overweight or have other risk factors.