Yup, diet, and unfortunately, I think genetics.
I keep very low bodyfat naturally, whether I want it or not(kind of sucks), and almost always have at least a four pack, but I always have love handles no matter what my BF level is. On me its the very last thing to go. I'll have veins bursting through my pecs before those little fuckers are gone.