88qba88, amazing, thank you! Now, that you say it, it seems really quite obvious, but I didn't manage to figure it out myself. I'll do that an post the result.
In the meantime however I had a chat with the graphics department - they'd be happy to have the buildings be fully white, but still cast some AO shadows. Like the image attached, just with the buildings being white. Will play around to get that result.