Right, in the final render it will look correct, as displacement will be calculated for that particular view and then the view won't change. You could use World displacement instead, but that would use significantly more memory and pre-processing time, so not recommended (but then when you move the view, displacement would have been calculated for the whole scene so the new view will look correct also - quite a price to pay for just being able to move the view in IR, though, and is there for an option in other situations :) )