Make sure you have the latest version of SR4, which is 4.3.0.19614. Next, try recording Full Screen and see if that works, then chose (and recreate) your custom setting and see if that works.
If it still doesn't, turn of
Enable hardware encoding from the Preferences | Video menu and try again. Whatever you find you may want to report the issue to Cyberlink. Here's their contact
page.