I think it´s about automating things and not assigning stuff by hand every time. I'm a big fan of this, that's why I'm always try to use one keyed/animated cam (+CameraMod with all post processing settings) for all shots if possible. So maybe the key here is just to key the env map(s). I would try to do it like this:
Using CoronaMultimap as scene environment map, setting it to "Material", plug the Hdris (as corona bitmap) you need in the slots, key all "Frequency" to zero at frame 0, key slot 1 frequency (for camera 1) to 1.0 at frame 1, key slot 2 (for camera 2) to 1.0 at frame 2 and the rest to zero and so on for every shot which needs a different HDRI. You then have just to render the apropriate frame for the chosen camera without having to change anything manually over and over again.
Good Luck