Glass is set to hybrid... (Double glazing with thickness per glass)
Just had a look at the scene to make sure I didn't miss anything, without the portal I'm getting about 800k rays/s, with the portal about 650k max.
I flipped it both ways and tested to make sure I didn't mess up the normal direction, but slower regardless.
Keep in mind the scene is lit by bounced GI rays only. The sun hits the vegetation outside, which in turn lights the interior.
I just figured that's why it was a fairly slow scene and just let it cook overnight for 300 passes. I'd say it's almost noise-free around about 250, but still a fair difference with 300. :)