Bump always adds to render time, albeit 5 times sounds like way too much, but do you really need bump in this scene. I think glossy map should be enough, especially if it can save so much time. As for brightness, you can render the scene much brighter and change exposure afterwards. Again, if it helps to save time, then why not.