From the help file:
Turn off to have the mental ray renderer correctly render height maps. Height maps can be generated by normal mapping; see Creating and Using Normal Bump Maps .
When using only height maps in the scene, make sure this option is off. If some objects in the scene use height maps while others use standard displacement, apply smoothing on a per-object basis (see mental ray Panel (Object Properties Dialog) ).
When on, mental ray simply smoothes the geometry using the interpolated normals, making the geometry look better. This result, however, cannot be used for height map displacement because smoothing affects geometry in a way that is incompatible with height mapping.