If going for the evening crossing from Toulon, you'll have plenty of time to do a full lap of the Gorge, including the outstanding D23 loop on the North bank, and still take a good cross-country route. It's a comfortable morning ride to do both sides of the gorge as long as you're not stopping every 100m.
This route would be good but might be a touch too long – though you could shorten it by using some autoroute in the afternoon - you could play it by ear and adapt if running short of time.
However, if you're only riding one side (and as everyone has already said, the southern side is the most spectacular - it includes a stretch called the "Corniche Sublime", which more-or-less means "excellent ledge") I wouldn't bother going all the way to Moustiers St Marie - just head south to Aups once you've descended to "ground level". It's slightly circuitous... but I'd go
this way GMap says it's 5.5 hours, but I'd guess it's more like seven.
Edit: fix link