Displacement is either calculated by "screen" or "world" - when screen, it is based on what is included in the camera FOV, to reduce having to calculate displacement for things not seen in the camera. It is only calculated once at the beginning of IR, so changing the FOV can bring into view things where displacement is not calculated to as much detail, or not calculated at all. Recalculating it would of course slow down IR, which leaves the option of swapping displacement to World (with the penalties involved - it will likely take longer to calculate when you first run IR, and it will likely need quite a bit more memory).