I've always thought it was retarded to wait. Why the hell is playing some sort of game with someone cool?
If I gave someone my number and they didn't call for a couple of days (unless they were out of town or something) I'd think "fuck that". If someone isn't into me enough to call me, screw them.