Not necessarily. I have a group of girls doing 6 sets of 3 and taking normal rest between sets. Obviously, the bands still increase speed even with varied sets x reps and rest.
i dont really count my time between sets, the only time i take is that time it takes the guy who goes ahead of me to get up and for the weights to change. but you shouldnt be taking alot of time between sets.