I use motion jpeg and if I use the antishake on the file prior to rendering it's not bad, but does increase the chance that interlacing artifacts can result as per my other post. Also because it uses quite a bit of the top and bottom of the Video frame (to do the antishake), then there is some quality loss when it expands the remainder to fill the frame, but not as bad as when applied to heavily compressed output.
I think the main quality problem for applying it to AVCHD is that it expands the image fields to fill the frame so it can use the top and bottom portions, then re-renders the output. In a sense kicking a man when he is down.
I suspect there is little that they can do and your best bet is to apply the antishake on material prior to compressing it to AVCHD. If your camera is AVCHD then you have no real choice unfortunately..
Intel Quad Core, NV 8800GT, Vista (32bit), Canon Powershot TX1. Editing 1280x720 (p) NTSC 30 fps motion jpeg in .avi container....to AVCHD DVDs for Blu Ray playback