You made the right decision to pull out if you weren't comfortable.
However, I will say that leading a tour is extremely difficult especially if you aren't sure of the ability of everyone. Generally most riders are good and understand the reasons for riding at certain speeds or having to cover large distances but you will always get one who says it's too fast and one who says it's too slow. It's not a race but equally you can't stop everywhere to see every single interesting thing but a good tour leader will try and get the balance right. What is an easy day for someone can be an Iron Butt challenge for someone else.
Before a trip it's always good to speak to the guide to try and understand what speed he rides at, daily distance travelled and road conditions as you'd be surprised at the number of people who don't know this before setting off!!
For instance, on my Alaskan trips we will cover over 250 miles on some days because everywhere is a long way from everywhere else. Some people find this a problem until I speak with them because it's not 250 UK miles with heavy traffic and windy B roads or Alpine mountains. I explain that if we ride at 65mph, we actually do 65 miles per hours so over a day, 250 miles equates to about 4 hours easy riding which gives plenty of time for picture and coffee stops etc. It makes a huge difference when people know this.............!!
Over the years I've taken some very inexperienced riders to places they never thought they would get to and they've survived and enjoyed every mile. The satisfaction to me is huge
On my trips, I will always ride the stated route at around the posted speed limits but people have the option to either ride ahead or ride a little slower if they like. They have the route and the destination motel so generally this works well. I do like to keep the more inexperienced riders close to me though until they start to get confidence as riding in the middle of nowhere in Alaska can be very intimidating.
The other big issue is group size and I find 8-10 ideal and manageable as you can keep an eye on everyone.
Saying all that, I've done 15 guided trips all over the US and Canada mostly with around 140 tossers from this august institution and can count on one hand the number who didn't like how I did things with regard to group riding.
Guided trips aren't for everyone but they work for a lot of people. My advice would be to investigate the tour company a little deeper, talk to them and maybe previous customers so you get a full picture before you go.