I don't think this can be achieved.
I just tried with CoronaAO (set to "calculate from inside") as the "Radius map" for the rounded edges map. You can then specify "Ray directionality" in the AO to exclude some of the edges, however this method is not very easy to use, and produces some unexpected/hard to understand results.