That should be quite easy to test. Move your camera closer to the brick wall and render small crop. If bricks still looks bad and jaggy, then it's probably due to low quality/resolution of displacement texture. However if it looks much better at close up, then most likely that means that displacement screen size is too big. If i'd have to guess, i would say it's latter. In your attached example, gaps between the bricks are about 2-4 px size, that means at default settings they are subdivided just in 1-2 triangles - that's nowhere near enough for good looking displacement.
I would probably leave displacement for close-up shots and for distant ones i would use normal map. The gaps would look much cleaner.