I'd say this is expected. If you turn off the Sun, it will no longer contribute direct lighting or shadows, but it is still being used for the Sky to know "what time of day it is". If you want to disable that aspect of the Sky, you could create a second Sky not linked to any Sun, or to a different Sun for the specific time of day.
EDIT - or you can switch the Sky to manual, and "Clear" the Sun selection to get a sky unaffected by any sun.
