Essentially, I know of 3 ways to do it (aside from the obvious splitting the object apart).
First, you can use polygon selections directly. Even with highres and high quality textures you will likely get artifacts around the edges of the selection. It's something you cannot change, just soften a bit by increasing the displacement quality (lower screen size / world size), but you will need much more RAM and it takes longer to render. Also blurring the displacement texture does not help in this special case. It's not a bug. It's just the way it works. For some "noisy" displacement on a wall, it might even be fine enough.
Second, you can mask the displacement area out with a vertex map, a smooth operation this is. Depending on the vertex count and distribution of your model, this can work great or insufficiently. More and more densely distributed vertices are better here.
Third, there's the good old black and white mask. You got to paint one, but with some smooth edges you will most likely get the best result.
Don't use the "old" (non-2.5D) displacement with polygon selections, it will tear apart your mesh at the edges of the displacement or if you trick this away, it will still cause artefacts.
Maybe somebody else has another idea, I am all ears!
CU
Michael
