I don't understand why would you ever care about rays/sample. It's as irrelevant attribute as it gets. Rays/sample depends on how complex your materials are shading-wise, but if they look the way you wan't them, you just can't do any compromises in quality. You can't just decrease rays/sample count without dumbing down your materials other than worsening their quality, and worsening quality is never a solution.
So i would first try to just assign gray diffuse material to everything, to see if the problem is material related, or if it's lighting related. Also, portals, when placed wrong may cause extreme slow downs.