Unless I get injured and require medical attention, yes. If you can put them on the bar and lift them, you can take them off and put them back.
BTW, I once tapped Rickson for leaving 2.5s on the EZ Curl bar. Of course, he was at the water fountain and I hit him over the head with a 45 from behind first. All technique after that though.
I always put them back because it perturbs me when I gotta go across the gym to fetch something. If I use it, I replace it. Including if I am using it after someone left it there.