Twizzle
All I can say is I've never removed the silencers or dropped the swinging arm by removing the lower shock mounting bolts.
However....you do need to drop the silencer/s a bit to get the spindle undone and out so it could be easier to take the silencers off - if you can, as sometimes they get really stuck on and there is much hammering and graunching needed to get them off. DAMHIK

It should be enough to take out the rear two bolts and let the silencer drop if you can't get the appropriate one off. The spindle can enter from either side, although it usually enters from the right.
I can see that it would be easier to drop the swinging arm especially if you are using a centre hoist. I've got a bench with a hole the rear wheel drops through. If you don't drop the SA then I'd use a trolley jack or something to hold up the wheel while you get the spindle out and manipulate the belt and brake assembly off. The more I think about it the more I see why dropping the SA is a good idea, but I've never tried!
It could be a good idea to have a helper to hold things in place as everything weighs a fair bit although given your size you could probably do it all with one hand tied behind your back, I'm just a 13 stone 5'11" midget and I manage
Also it would make life, and reassembly, easier if you take the calliper off the bracket although I never have. In fact, if you drop the SA you may have to so as not to strain the hose - can't be sure about that though.
The more stuff you remove the easier the job will be from an access point of view.
The other thing I do is repack the wheel bearings when I take the wheels off but I am a bit anal about maintenance. They are tapered rollers on the older bikes and you have to take the outer seals out to get the bearings out. All easy stuff when you know how. Most people don't bother but you could just check there is a bit of grease in there, but not too much!
Hope this helps.
Disclaimer - this info is just from my experience and recollections. I may have forgotten stuff....
Cheers
Dave