I would definately take them off, not because they unbalance the wheel (TBH I had mine on for months and apart from the constant whistling, couldn't tell the difference) but because someone may nick them. My one disappeared (who knows if it worked itself loose, or if it was pinched) but either way it bloody annoyed me as it was a really useful bit of kit. Cost me £5 IIRC for a very solid brass looking one.
Seriously, if they only come in a pack of 2 and you don't want 2 in your toolbag I would happily go halves with you.