While this definitely does need fixing, adaptivity will always need to be evaluated for background as well. BG texture needs to be antialiased, otherwise it will come out jagged, jittery, or very noisy when in DoF or Motion-Blurred.
As a temporary solution, I would suggest to set noise limit to 0, and use pass limit instead. This way, image will still be evaluated adaptively, but won't stop prematurely. You will just need to do some test to see what pass limit is about right...