This is an intended behavior. Displacement is following the direction of averaged normals. I understand that the behavior that you would like to see would be displacement being always perpendicular to the surface, right? So for example if we create a box with just 6 faces (no additional subdivisions), each face would be displaced exactly in the direction of each of the faces' normals.
But if we do this, what would the corners of this box look like? :)