I ran into this problem just the other day.
I used a plane a good distance away from the buildings, a place a cloud bitmap on them. For a BG and to cast reflections. The problem i had was no matter what material or settings i tried, it always changed the lighting.
The first picture was render with the cloud plane.
The second picture i rendered without the cloud plane, and used a material id/alpha to mask in the clouds and reflections on the windows.
Notice the huge difference in the lighting and shadows.
Things I tried to get it to work in the render.
Corona Material with a variety of combinations, Translucency, Opacity, and Refraction.
Corona Light Material with and without Emit Light on.
I even excluded the plane from casting shadows.
Everything resulted in the place changing my lighting and shadows. It however did not increase my render times and noise.
My advice try a render without the Background Plane and see if it clears it up any. Maybe even try a render without the glass too.