I always find the number one issue with, or rather the first way I can tell a brick exterior is not realistic, is by the coursing. When bricks/mortar joints don't align where they should it throws the image off. The second biggest issue I notice or struggle with are corner and window returns. Thankfully sills/headers can be hidden by stone sills/lintels, etc, but sometimes this isn't in the design.
I normally use bercontile for my brickwork to get variations/non-repetition, but a good bitmap with proper displacement and additional layering of dirt/marks and noise also works well as suggested already.
I saw someone suggest using Rayfire for bricks recently since it offer good controls over brick variation, but I haven't been able to test it in terms of memory consumption. Seems doable for a small/normal sized exterior, but when you get into a massive facade I'm not so sure.