Noise being greater with an HDRI vs Sky may not be surprising (the Sky is fairly even lighting, while an HDRI can vary greatly in lighting from specific directions unless it's a clear blue sky :) ). The question is, are you getting more noise with an HDRI with portals, vs an HDRI without portals. That shouldn't happen - but "more noise with an HDRI rather than sky" is reasonable I would say, depending on the HDRI in question.
Let me know which one it is :)