It really comes down to the specific scenario. If you have for example strong light hitting quite reflective surface that has a bump map, and it's hitting it in the exact incident angle of the area you are rendering, then rays bouncing off the the noisy objects are noisy because they are randomly hitting surface of very different brightness. Some rays will hit the flat part of the bumps, and get average brightness, some rays will hit shadowed part of the bumped surface, and return very dark color, other rays will hit highlighted part, and return very bright color. It's lot less predictable than flat surface without bump.
Non the less, it would be good to post at very least picture of the problem, so we could make sure it's not some bug, or something that could be significantly improved by changing of some setting.