ok, problem solved (or, rather just explained). The noise is not from direct illumination, but from indirect one. You can confirm by checking the render elements. The problem is that the burnt wall spots when light is close are themselves a source of illumination for the rest of the scene. You can control the noise by lowering MSI.