I have noticed for at least 2-3 years that Handbrake gives incredibly good quality to compression ratio. File sizes are basically half or less than a normal h264 encode using constant or variable bitrate. The secret seems to be its use of "Constant Quality" and a "Rate Factor" sliding scale. For instance, I generally use a rate factor of about 20 on DVD or other SD resolution sources. For 1920x1080 I usually use a bit more compression (higher RF number), somewhere between 21 and 24. Animation in 1080p, for instance, looks very good at 24rf with little compression artifact - and the file size is amazingly small.
SO the problem is: PD 11 simply does not offer rendering using the Constant Quality or Rate Factor method or anything nearly as efficient. You are limited to specifying bit rates of 13000 to 26300 kbps on all h264 encodes. This is simply not able to achieve quality vs filesize that I am seeing with Handbrake encodes - again file sizes out of PD 11 are about 2x what is achievable.
Now understand that I am editing almost exclusively AVCHD, currently 24p footage from a Sony NEX 5n, and some 60p.
I actually don't do a lot of effects and try to keep color correction to a minimum, I am mostly just trimming scenes and assembling and puttting some simple titles on, adding an occasional still slide show with pan and zoom. So I am able to do an SVRT intelligent render and save a lot of time, and at the same time get virtually no generational loss on most of my renders. SVRT only re-encodes where necessary. For my use this results in an output movie with a .M2TS format. I consider this as a 'master render' because, as I say, it is made up typically of a high proportion of absolute original quality footage. The SVRT rendering is actually one of the big selling points of PD to me, in fact.
Then I open this file in Handbrake, and do a re-encode to MP4 or MKV format, using the rate factor h264 method. Typically for my 1920x1080p24 projects I use RF22.
So the result on one approx 5m minute long movie project was:
PD 11, SVRT, .M2TS FILE: 652MB
PD 11, SMALLEST MP4 FILE: 410MB
HANDBRAKE, RF22 MP4 FILE: 239MB
Now, I am sure that you might be able to pick out some quality differences between the .m2ts file and the Handbrake file. But to my eye it is small and perfectly good looking. Moreover, the difference between the two mp4 files is even smaller. But that is my eye and of course YMMV.
Now, if for whatever reason you can't do an SVRT intelligent render, then I would try just rendering to the absolute max quality you possibly can in whatever format and using that for the Handbrake re-encode. Try different rate factors if you want to experiment with the quality vs file size see-saw.
So Handbrake is free software and I have used it for years. It's a bit of a mystery why more commercial video tools like PD and other editors don't simply switch to CQ rendering; it surely would be nice to at least have the option somewhere, rather than having to open a completely separate app.
Also, I would certainly love it if PD adopted another neat trick from Handbrake's arsenal: you can set up a whole bunch of renders of different movies, or the same movie to different resolutions and rate factors and file sizes etc, in a queue, and then hit the go button and leave your computer to grind away for hours, and it does each job in succession. You can even add more jobs to the queue even as one render is taking place. So cool. Handbrake is really amazingly good.