I took mine off for a while.
Didn't feel any difference but my concern was dropping it when playing off road.
As this playing is usually more that 100miles from home on a weekend spin I though I'd put it back as a precaution.
If you're not likely to drop the bike and get stranded, take it off. You won't really notice.
It's useful for attaching a cheap Hein Gericke Bar bag to anyway..