The Caustics render settings have an option to include or exclude objects, designed for this very situation (where distant objects are receiving caustics so they don't resolve cleanly due to how far away they are and really don't need the caustics on them anyway). Note it is done by object, not by material :)