It should only affect how the light is bouncing around, and if it gets darker, this likely means the darker is more accurate. Might be useful if you post examples though so we can see what you mean, in case there is any problem there :)
It's safe (and better) to use the 4K unless you happen to get square artifacts, which only show up in very rare niche cases (so, try it and use it, unless you find it gives you problems which is unlikely).