I am afraid this is too complex case to be handled by displacement. What you could try is blurring the texture (a lot), to get more smooth edges, but obviously that would affect the way it looks.
Other than that I can only recommend using separate objects for the bricks, instead of displacement.