I also believe RoundEdges just like AO don't work properly as displacement. That would be super cool if it did though.
Anyway, geometric solution to your problem OP could be using inset (before or after, don't know how the script operates), similar how game artists use it to avoid smoothing group issues with chamfers.