Don't change from default, this conflicts with Adaptivity. (In rare cases it may make sense to adjust AAvsGI, but since adaptivity, this is being done by a automatic multiplier for rays vs samples on a per pixel level)
Directional shows the cone edges of the light. That gradient of light requires by definition more rays to be displayed properly, however the difference should be small.
As such it suggests some other interplay of settings or the scene setup being the culprit. Post a screenshot or a scene, where you can reproduce the slow down and someone will take a look at it.