Hi Ovenpaa:
The mounting balls on the two RAM components I use (one to hold the Garmin bracket, the other attached to the handlebar) are still in very good condition after 4 years and 100,000 km of use. I mostly park my motorcycle outside, where it is exposed to sun and weather.
There is some deformation in the surface of both of the balls as a result of the extension clamp always being in the same place, but certainly no evidence of any bits of rubber coming off. The deformation in the surface doesn't seem to interfere at all with the functionality of the mount.
Perhaps take some good quality, close-up pictures and send them to RAM Mount in the USA. My guess is that they would post you replacement components if it appears there has been a failure of the rubber.
Michael