I suspect "light samples multiplier" 16 may be causing some troubles. Set it to 1 or 2 and try again.How do you make your portals? It's best to create a single face capping the window hole. Also read this:http://corona-renderer.com/wiki/performance_debugging
My bet would be curtain material :)