The way you plugged glass material into rayswitcher, effectivelly disables the latter material. You need to unplug glass mat from rayswitcher's global illumination slot in order to exclude that material from GI.
Also, you should disable diffuse component on glass material, by turning diffuse level to 0 or by setting its colour to black.