My test, with default parameters...
How many passes? Also, I think the biggest issue is when there are objects smaller than 1 px (like small branches) and there is subtle movement.
ok what is the values you use for sharpening & blurring? also default?
I've just done 1 render with 200 passes and it is still grainy, but takes an hour a frame ...:(
For sharpen/blur I used the default values.
You should get roughly the same render time with:
100 passes and gi vs aa set to 16
200 passes and gi vs aa set to 8
400 passes and gi vs aa set to 4
So if you set passes and gi vs aa value proportionally, then the render times should not rise.
You can also try setting light samples multiplier to 1. It should not affect render quality in your case, and will additionally cut render times.