One more thing came to my mind - if you have adaptivity enabled, it could slow down cleaning picture with objects such as tree(s), grass, etc. Because there are lot of noisy points between leaves, grass blades, etc. which make difficult to make picture equally noisy all over the picture's area (not really good explanation, you should read more about adaptivity). But in fact you'll not see this noise because of amount of small details in tree's crown or grass field. At least in my exterior scenes I always disable adaptivity to speed up rendering, but I always keep it enable in interior scenes.