Denoising - AVX version question - kinda urgent

[alexyork]

Hi devs,

We are running into an issue with some of our 128-thread Threadrippers, where during Corona High Quality denoising it's pushing temps up above the safe threshold and shutting down the machines.

Running Corona v6 hf 1 - which version of AVX are you guys using currently? These machines are currently overclocked and tuned for avx2 - so if you're using anything else like AVX512 this might be why.

If you could ideally let me know asap as we're having a new machine built right now and they might be able to tune it differently for us.

Cheers,

[Juraj]

Since Threadrippers (or AMD chips in general) don't have any AVX512 instruction support (since it has to be physically on die, only Intel Scalable Xeons have it), Corona would utilize the CPU 100perc. no matter what tiny subset of instructions were injected into code during compilation.
We know that this is something the devs can't get rid of, I asked for it for years since it affected the boost clocks of my Xeon farm nodes. The culprit is Microsoft's compiler.

Threadrippers only support AVX2 set.

The only solution is to not run fixed voltage overclock on Threadrippers. It's incredibly dumb idea, no matter what some system integrator believes. People who absolutely must overclock Threadrippers, can do so using custom voltage curve and power limits (through so called "PBO 2", recently introduced last year).

[alexyork]

Thanks Juraj.

Corona Devs - any clarification from your end on this? Just need to know for sure if v6 hf1 uses AVX512 in anyway (even lightly).

Many thanks,

[Ryuu]

Corona itself does not use any AVX at all. The highest we use is SSE4. We're using the F16C instruction set when available, which can technically be considered a part of AVX and TBH I have no idea if just using this might cause the CPU to switch to the AVX clock or not. But we use these instructions only during rendering, not when computing denoising.

We do however have some dependencies which use AVX:

• Embree uses AVX and AVX2 when available. We can turn this off if needed.
• The runtime library of the compiler we're using uses AVX(2) for the implementation of some functionality. This is something that is completely outside of our control.

That being said, like Juraj already noted, no AMD CPU at this point supports AVX-512 and we cannot execute instructions which are not supported by the CPU. So even if we had some AVX-512 code, all that could ever cause on an AMD CPU would be to immediately crash Corona due to executing an unknown/unsupported instruction.

The denoising is probably the only part of Corona which is not limited by memory accesses and therefore really utilizes the CPU as much as possible - from our CPU FAQ:

Why is my CPU temperature higher/fan louder during the denoising compared to the rendering?
The memory access pattern of the rendering code is very random - for example, each ray may hit a completely different object. This makes it very hard for the CPU to predict which parts of the memory will be accessed next, and in the end the CPU ends up waiting for data to be fetched from memory most of the time.
 
When the image is denoised, it is accessed sequentially one pixel at a time. Also the processing of each pixel takes long enough so that the CPU is able to read the next pixel from memory before it is needed. This is a best case scenario for the CPU - it does not have to wait for memory, and the execution units are (almost) fully utilized.
 
When the execution units sit idle waiting for memory, the CPU is able to optimize its energy consumption (and therefore heat generation), but when the denoising kicks in, the CPU units are kept busy all the time and so the temperature goes up.

So the most probably culprit is just an unstable overclocked configuration, which runs just fine until the CPU is under heavy load.

[alexyork]

Thanks Ryuu! This is really helpful. It does look as though our overclocks are tuned aggressively and didn't allow for any headroom for denoising, causing the issues. I'm sure we can get them adjusted.

Do you think disabling Embree would make any difference? Would that cause issues elsewhere within Corona/efficiencies?

Cheers,

[Ondra]

Thanks Ryuu! This is really helpful. It does look as though our overclocks are tuned aggressively and didn't allow for any headroom for denoising, causing the issues. I'm sure we can get them adjusted.

Do you think disabling Embree would make any difference? Would that cause issues elsewhere within Corona/efficiencies?

Cheers,

Starting with v7 we no longer have an option to run without embree with practical performance - there were too many holes, missing stuff, and inefficiencies with the backup structure, that we decided to remove it. So that is not a viable solution... also embree does not really influence denoising.

[alexyork]

Thanks Ryuu! This is really helpful. It does look as though our overclocks are tuned aggressively and didn't allow for any headroom for denoising, causing the issues. I'm sure we can get them adjusted.

Do you think disabling Embree would make any difference? Would that cause issues elsewhere within Corona/efficiencies?

Cheers,

Starting with v7 we no longer have an option to run without embree with practical performance - there were too many holes, missing stuff, and inefficiencies with the backup structure, that we decided to remove it. So that is not a viable solution... also embree does not really influence denoising.

Makes sense. So Embree will always be enabled and in use if available?

I have passed all the info above to our system provider. Hopefully they can retune some things to factor in this overhead. To be honest, it would be SUPER helpful if you guys could update the Corona Benchmark to allow for longer renders with denoising added. It could then be used as a simple way to test stuff like this (if not a proper test).