Pokoy & Corona_GK gave you one alternative, RoundEdges (with big radius, and don't be afraid to use higher value for map than default 1). Pretty outside of box solution.
One with bigger control but less automatic is from Romullus, to use Vertex painting.
Most complicated is to do a second UVW channel for your mesh, do a quick random unwrap, and bake down your AO into texture for that second channel and you have your mask to drive displacement.