Sometimes, adding a chamfer modifier to the distance object helps with similar problems in the Distance Map. I've seen problems that were caused by overlapping meshes that I couldn't resolve, though. You may have to remodel overlapping objects to be one continuous geometry (or use boolean), this should help.
The Distance Map has a number of issues, this is only one of them. I'm planning to post an extensive problems/request thread about this map with the intention to get our beloved developers to make it more robust and flexible but didn't have the time to do so. Hopefully soon.