I've seen the issue and when I was making a screen recording it disappeared. Try this: When you first open the Pask Designer, place the first rectangular mask like this:
Then place another mask, and when you adjust the feathering it will only affect the inner edges of the second mask instead of affecting the outside edges of the frame.
On my test system, I didn't even save these changes and yet it works normally now and I can't get the original issue to appear. You may also need to turn of OpenCL under hardware Acceleration, but once the mask is working as expected it doesn't seem to matter if that's enabled or not.