It is not glossiness. It is that when smoothing is off, those flat polygons reflect only very small portion of HDRI, and since your HDRI does not have very high resolution, it is filtered, and looks blurry.
Basically, when smoothing is on, polygons reflect quite wide angle of the environment. While when smoothing is off, polygons reflect only area that is perfectly perpendicular to the surface.
I am also attaching picture to better explain