xeon v4 cpus

[peterguthrie]

Does anyone have a machine with the new v4 chips yet?

We are encountering some weird issues, like not being able to set affinity in task manager, and also 'Runtimebroker.exe' going bonkers and using up one core all the time.

thanks for any help

Peter

[Juraj]

Hm, no idea yet, could be related to processor groups since these have so many logical cores.

But anyway, something quick to try, do you run the latest Bios/Uefi ? Sometimes motherboards arrive on market sooner than final CPUs.

Also what Windows is this ? And does it have all the latest updates ?

[peterguthrie]

Hi Juraj

windows 10 pro, all windows updates installed. The people who put the computer together assure me the bios is fully up to date

[Juraj]

Can you better describe "not being able to set affinity" ? At what step does the issue occur ?

[peterguthrie]

as soon as you click set affinity, you get the error attached

[Juraj]

Oh ok, this should be classic Windows elevation issues.

You could either escalate Task manager by clicking "show processes from all users", or typing "r"(run) and then "taskmgr ". Alternatively, you're running an application that running with Admin rights, so you could un-check that.

[hrvojezg00]

I have the same problem, seems to be Windows issue. Only workaround is setting the # of threads in Corona Performance settings to the amount of cores you want to use at the time. I think this happens with computer with over 64 cores.

[FrostKiwi]

Had a similar runtime broker issue after installing win 10, non xeon cpu related issue. This helped:

Although, I ended up uninstalling half of windows 10 including cortana, search indexing etc., so I cant recreate or help.
Hope this is not bad info.

[Juraj]

I have the same problem, seems to be Windows issue. Only workaround is setting the # of threads in Corona Performance settings to the amount of cores you want to use at the time. I think this happens with computer with over 64 cores.

Is your UAC completely turned off ? (should be). And do you use elevated task manager ?

[peterguthrie]

UAC slider at bottom (can you turn it off even more?) elevated task manager - same story.

@sairesart already turned off all notifications, even uninstalled all 'metro' apps from windows 10.

runtimebroker still constantly uses 2.6% of the cpu which means the whole of the windows 10 UI feels a bit sluggish.

starting to suspect windows 10 just doesnt work well with these new chips and its simply a problem they need to fix.

[peterguthrie]

maybe i should turn off hyperthreading and see if windows 10 prefers having less than 64 logical processors???

[Juraj]

You can only disable UAC through registry, and it yields some drawbacks.

If this doesn't correct the issue though, I suggest to take it back. (I think this step requires restart to work...or not, not sure)

[peterguthrie]

pretty big performance hit disabling hyper threading sadly (on the benchmark i went from 39s to 55s)

i havent had any runtimebroker nonsense since though...

would be VERY annoying if this is an issue with windows 10 and new xeon chips with more than 64 cores....

[peterguthrie]

ps. Juraj, i check the bios and its the latest one (which supports the e5 2697 v4 )

[hrvojezg00]

We have an e5-2690 v1 system (no problems with setting affinity on 36 cores) and e5 2699 v3 (problems with setting affiniti on 72 cores). I`m sure its windows limit issue.

[alexyork]

ps. Juraj, i check the bios and its the latest one (which supports the e5 2697 v4 )

Hello mate. I spoke with Kern over at Scan about this a while ago and he mentioned that there were BIOS issues with these new chips, so we decided to hold off buying one. I would guess that this is the problem and that an update is needed from the mobo manufacturer (asus?). Although I imagine you've already been onto scan about this. We're thinking of plumping for the same chips you did so keen to find out what the problem is!

[peterguthrie]

Hi Alex,  feels like this is a Windows issue rather than a motherboard driver issue. 3dsmax and corona work great, it's just a couple of annoying things with win10

[alexyork]

Hi Alex,  feels like this is a Windows issue rather than a motherboard driver issue. 3dsmax and corona work great, it's just a couple of annoying things with win10

Ah right. Glad those two are working properly. That's what matters most I guess.

[Ondra]

Hi Alex,  feels like this is a Windows issue rather than a motherboard driver issue. 3dsmax and corona work great, it's just a couple of annoying things with win10

So refreshing when just for once it's not *our* problem :D

[peterguthrie]

anyone else got one of these new chips yet? curious to see if you have the same problem as us?

[Juraj]

None of the people I've talked to. Will have such build in 2-3 weeks so I'll see as well :- ).

[Jann]

moved to separate thread

[peterguthrie]

the point of this thread really was about windows 10 having issues with xeon v4 cpus with more that 64 cores - i'm sure no-one would mind you starting a new thread

I'm still very curious to see if anyone is having a similar experience as Henry & I?? we surely can't be the only people to have these machines??

[Juraj]

Which motherboard do you have ?  Let's try it from this way. Even if you have Asus, there are 4 versions (D8/D16, WS, Non-WS, L), so there can still be some local culprit in the bios setup.

(And slightly irrelevant a bit, could you send me your full build into Facebook message ?)

[maru]

Unfortunately this indeed seems to be Windows problem, and we had few (maybe 2?) cases like this in support. For now the solutions are:
-disabling some threads in Corona settings
-trying to change affinity BEFORE doing anything in 3ds Max - strange enough it looked like Windows allows you to change affinity before you try some more advanced tasks like rendering (even with scanline) - I might be wrong here, so it would be great if someone could check this (changing affinity right after starting 3ds Max, without doing any complex tasks like rendering)

This is completely ridiculous as it doesn't allow the users to really utilize the power of their CPUs.

Update: but generally, Corona works 100% fine on 64+ thread systems i 99% cases! This is confirmed by our tests, and by user tests. The problems only happen in some specific cases.

[Ondra]

remind me, what was the issue exactly? I remember some about Windows running 100% utilization on some cores, but that could be hardly fixed by changing number of threads inside Corona...

[PROH]

Just saw this answer in the Area regarding CPUs with more than 64 cores:

"Windows 10 supports maximum memory configurations that were formerly available only in the realm of servers. Windows 10 Home supports up to 128GB of RAM on the x64 platform. Windows 10 Pro and Enterprise both support up to 512GB on the x64 platform. The x86 versions of Windows 10 support a maximum of 4GB of RAM. Windows 10 supports a maximum of two physical CPUs, but the number of logical processors or cores varies based on the processor architecture. A maximum of 32 cores is supported in 32-bit versions of Windows 8, whereas up to 256 cores are supported in the 64-bit versions."

So it seems that 64 cores should not be a problem.

Here's a link to the whole post:
http://forums.autodesk.com/t5/3ds-max-3ds-max-design-general/cpu-rendering-core-thread-limitations/m-p/6389840#M115272

[peterguthrie]

the problem is how windows 10 deals with new processors with more than 64 cores.

1. Windows 10 has some annoying issue with 'runtimebroker.exe' that we STILL havent been able to fix despite trying every supposed fix online. the result is restarted our machine every couple of hours. It means windows become very unresponsive and laggy.

2. When rendering, we can't change the affinity in task manager like we used to be able to do. This is what maru was referring to above, so at least we arent alone in this!

Both are very annoying and there doesnt seem to be any recognition of the problem in microsoft-land

[Juraj]

I have friends with 80 threads on Windows 10 Pro/Enterprise and they don't have this issue, so there must be some solution.

I am about one week from completing mine (also 40 cores/ 80 threads) (I am playing with water loop so it's taking bit of time), and then I will finally see what this is about.

[Dalton Watts]

I have my 2x xeons 2699 v3 (actually they are 2696) with windows 7 and never had to set affinity for Corona. Vray 3.4 also doesn't need any cpu affinity "hack". The only thing i see is that they are slower saving/opening scenes than my previous single socket systems (i7 4770k and xeon 1230 v2) Never tried windows 10 with them though.

[Juraj]

Installed my new 80 thread machine, with Windows 10 pro (although I really should get brother's enterprise from school so I can get rid of all the forced crap).

Alright, some issues seem to resurface after reinstall ;- ) Spoke bit too early maybe.

[alexyork]

That's encouraging.

Peter, we're ordering one imminently so will let you know our findings. I wonder if you have some dodgy hardware or software issue that is unique to your machine because Scan are not reporting any issues. Did you have the same problem with the other one you've got?

[Juraj]

Alright, the affinity really is there as bug (but only for applications that call for it, so..yes, Corona too), but runtime broker only happens under particular conditions. Doesn't require restart, just restart the process :- ) BUT, annoying.

Another strange thing, Corona Benchmark only uses 95perc. of cpu, and underclocks. Cinebench works correctly (perfectly !!! :- )  ) Is this because of AVX instructions ? Broadwell has (for whatever odd reason) lower turbo with AVX sets. I don't get it either.

Last, we can all blame the fantastic Windows Processor groups (which Windows creates when more than 64 threads exist). What a fucking feature....

By default, an application is constrained to a single group, which should provide ample processing capability for the typical application

Thanks Microsoft, but no.

https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx

I believe some of this could be solved by newer bios too, but there would have to be one first...

[GabaCGStudio]

Installed my new 80 thread machine, with Windows 10 pro.

hi Juraj,

what's your case chasis platform? EEB or ATX or eATX?
and what's you fan? u14?

[alexyork]

Alright, the affinity really is there as bug (but only for applications that call for it, so..yes, Corona too), but runtime broker only happens under particular conditions. Doesn't require restart, just restart the process :- ) BUT, annoying.

Another strange thing, Corona Benchmark only uses 95perc. of cpu, and underclocks. Cinebench works correctly (perfectly !!! :- )  ) Is this because of AVX instructions ? Broadwell has (for whatever odd reason) lower turbo with AVX sets. I don't get it either.

Last, we can all blame the fantastic Windows Processor groups (which Windows creates when more than 64 threads exist). What a fucking feature....

By default, an application is constrained to a single group, which should provide ample processing capability for the typical application

Thanks Microsoft, but no.

https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx

I believe some of this could be solved by newer bios too, but there would have to be one first...

So Peter's not alone then.... dispite what Scan tell me! I will speak with them today about this and try to get to the bottom of it.

[Ryuu]

Another strange thing, Corona Benchmark only uses 95perc. of cpu, and underclocks. Cinebench works correctly (perfectly !!! :- )  ) Is this because of AVX instructions ? Broadwell has (for whatever odd reason) lower turbo with AVX sets. I don't get it either.

Yes, this is normal. All Intel CPUs lower their max turbo frequency (or at least are supposed to) when using AVX instructions. The AVX execution units are usually switched off to save power and when they are switched on, the CPU disables the highest frequency bin to avoid possible overheating. The latest Broadwells are the only ones which can do this on a per-core basis. With older CPUs whenever any core was executing AVX, all the cores within the same physical CPU switched to this lower max frequency mode.

[Ondra]

Yes, this is normal. All Intel CPUs lower their max turbo frequency (or at least are supposed to) when using AVX instructions. The AVX execution units are usually switched off to save power and when they are switched on, the CPU disables the highest frequency bin to avoid possible overheating. The latest Broadwells are the only ones which can do this on a per-core basis. With older CPUs whenever any core was executing AVX, all the cores within the same physical CPU switched to this lower max frequency mode.

[Juraj]

That's super problematic, because it lowers them by 500mhz !! : / They still super fast, but nowhere near what they can do. I wonder if there's something that can be done.

Corona benchmark lowers all of them. I'll see if actual rendering does the same.

hi Juraj,

what's your case chasis platform? EEB or ATX or eATX?
and what's you fan? u14?

SSI-CEB. It aligns with E-Atx you just put away the standoffs that don't match (two of them).

Yup, Noctua NH-U14S, it fits even when all 16 slots are populated, I tried :- ) There is no reason to go for anything smaller if this fits so perfectly. The heatsink is very tall and the heatpipes have more than 1cm of space to the nearest memory slot. The fan also clears the Mosfet heatsink also. You could actually populate it with 4 fans( and set them to 500rpm for example even in 100perc.) if you wanted.

[alexyork]

I'm wondering if the E5-2697A is a solution of sorts, here. It's 16-core, so that = 64 threads across the system.

E5-2697A 2.6ghz * 16 cores = 41.6ghz
V
E5-2697 2.3ghz * 18 cores = 41.4ghz.

So on paper the cheaper CPU (£130 cheaper per chip!) is actually marginally *faster* than the more expensive one, will have a faster per-core performance which is better for Parsing and Displacement etc. and should also not have any issues with this Affinity problem.

Unless the issue remains when you are AT 64-threads as well as > 64-threads?

[Juraj]

E5-2697A is 140 dollars more expensive, not cheaper ( $2891.00 MSRP for A version).

Also, these mathematics need to be done with all-core turbo. But even that doesn't matter anymore if AVX sets underclock : /

E5-2697A  (turbo 5/5/5/5/5/5/5/5/5/5/5/6/7/8/10/10 ) so (2.6+0.5)x16= 49.6 Ghz
E5-2697    (5/5/5/5/5/5/5/5/5/5/6/7/8/9/10/11/13/13) so (2.3 +0.5)x18=50.4 Ghz

Also, similarly, they both have same single-core turbo of 3.6Ghz, so there should be no difference if they use it properly. But in cases like this, the one with higher base clock, will always be better for Workstation.

So the difference is marginal and will depend on unpredictable factors. I considered Cinebench R15 to be ultimate measure but the AVX crap thrown this idea away yesterday.

The groups should start after 64, but I will not put my hand into fire for it :- )

[alexyork]

E5-2697A is 140 dollars more expensive, not cheaper ( $2891.00 MSRP for A version).

Also, these mathematics need to be done with all-core turbo. But even that doesn't matter anymore if AVX sets underclock : /

E5-2697A  (turbo 5/5/5/5/5/5/5/5/5/5/5/6/7/8/10/10 ) so (2.6+0.5)x16= 49.6 Ghz
E5-2697    (5/5/5/5/5/5/5/5/5/5/6/7/8/9/10/11/13/13) so (2.3 +0.5)x18=50.4 Ghz

Also, similarly, they both have same single-core turbo of 3.6Ghz, so there should be no difference if they use it properly. But in cases like this, the one with higher base clock, will always be better for Workstation.


So the difference is marginal and will depend on unpredictable factors. I considered Cinebench R15 to be ultimate measure but the AVX crap thrown this idea away yesterday.

The groups should start after 64, but I will not put my hand into fire for it :- )

The A variant is definitely cheaper than the non A-variant here in the UK right now - £130 cheaper, per chip. Unfortunately availability seems to be very bad for this one...

[alexyork]

https://www.reddit.com/r/Windows10/comments/3fe1lx/runtime_broker_cpu_usage_fix/

Anyone tried this??

And this:

https://www.itunity.com/article/windows-10-runtime-broker-slow-performance-solved-2581

[Juraj]

I tried all of them yesterday :- ) You can guess the result...

[peterguthrie]

yep, tried everything I can find online, including both of those, no luck.

a proper fix from microsoft needed, and needed quickly!

[alexyork]

Well that's some BS right there.... I've asked one of the guys at Scan to urgently contact Asus for information in case this is an issue that can be fixed with a BIOS update.

[Juraj]

What abou the AVX instructions ? Is that something that has to be tweaked by bios or can Corona do something ?

My IvyBridge v2 Xeons, run at 100perc, full all-core turbo.

My Broadwell v4 Xeons, run at 95perc (all of them), and underclock even bellow the base clock (like negative all-core turbo heh). What's up with this ? That's more than 5-7 perc. loss of performance. Funky.
(this is only with Corona benchmark, other benchmarks run at 100perc. and correct turbo clocks).

But if I compare the rations between this benchmark and Cinebench, they are pretty similar...which leads me to question if it's only potential loss. So the AVX instructions are faster, thus don't need to run that high clock to provide identical performance.
But if that is how it works, why even bother introducing them ? To run at lower TDP ? There is no actual performance gain.

I don't get Intel, they fail to introduce any meaningful performance upgrade. V4 are only faster because of massive core count, but the per-core improvement is zero. The difference between v1 SandyBridge-E and v4 Broadwell-E is almost non-existent.
Quite a failure compared to the GPU improvements.

[Juraj]

Would it not be possible to compile non-AVX Corona benchmark ? Based on every reading, AVX does run on lower clock to fit within given termal limit, but it should still be faster overall. From my comparisons, it doesn't seem at all. Seems to be either the same, or little bit (2-4perc. slower). I guess I am asking for either stupid or impossible thing :- ) But my curiosity runs amok.

I now stand educated that there is such thing as AVX an non-AVX base clock, and AVX and non-AVX turbo bins. Like it wasn't complicated enough that Intel has 20 public and 10 OEM versions of CPU in each generation of Xeons family :- )

[Ryuu]

Would it not be possible to compile non-AVX Corona benchmark ?

We already talked about this with Ondra. We'll try to prepare something if there's time. The problem is that even if we have a non-AVX version of Corona, there can still be some AVX code executed from within the standard library, OS or even some other application scheduled on the same core. All it takes is just 1 AVX instruction executed each millisecond to keep the AVX execution unit powered up.

[Juraj]

I see. But since v4 can (as you wrote too) 'limit' only those cores that use AVX, it would still be interesting to see.

I thought that Vray would execute same behavior given they use Embree as default now too, but it didn't, so I presume they don't use AVX for it.

[Juraj]

For the others issues (if anyone is waiting), I am currently installing all Windows I have (7 enterprise, 10 pro, and Windows2012 server, might even try the 2016 beta), to see how each behaves.

The runtime broker in this cause is connected with the search function, that, and that only. But I've been trying for hours today to fix it, re-registering it,etc... to no avail. It's odd, can't for world see how it connects to existence of processor groups, but somehow, it does.

There is no newer bios for Asus boards also. It is strange, because the current one is from December, yet Asus has in marked as March '16 (when v4 came out), but it is the very same and I even tried overwritting it, it worked, but it indeed was the same.

[GabaCGStudio]

SSI-CEB. It aligns with E-Atx you just put away the standoffs that don't match (two of them).

Thanks for your reply ...

but two fans with one kilo weight (two kilos) in tower case?!?! i believe that the rackmount 4U case will be better for this config.

[alexyork]

SSI-CEB. It aligns with E-Atx you just put away the standoffs that don't match (two of them).

Tanks for your reply ...

but two fans with one kilo weight (two kilos) in tower case?!?! i believe that the rackmount case will be better for this config.

It's honestly fine. All our workstations are basically this exact spec and are rock-solid.

[Juraj]

but two fans with one kilo weight (two kilos) in tower case?!?! i believe that the rackmount 4U case will be better for this config.

Believe whatever you want, you're buying the case :- )

[Nejc Kilar]

SSI-CEB. It aligns with E-Atx you just put away the standoffs that don't match (two of them).

Thanks for your reply ...

but two fans with one kilo weight (two kilos) in tower case?!?! i believe that the rackmount 4U case will be better for this config.

Just want to put it out there in case someone doesn't know... While SSI-CEB is compatible with E-ATX (going by your word, I didn't check the compatibility list), SSI-EEB is not! If you have a SSI-EEB motherboard you need to buy a case that supports E-ATX + SSI-EEB.

Might be a total given to all the techies here but I reckon it is not like that for everyone...

Peace!

[GabaCGStudio]

Z10PE-D16 WS is EEB platform ... not CEB ...

[Nejc Kilar]

You've got good eye if thats the case! There are actually cases (computer chassis) that support both standards, you just need to move the standoffs appropriately. A lot of the cases however are E-ATX but have no support for SSI-EEB. You need to be careful to double check if it really supports that particular standard.

I am unsure how things are with SSI-CEB with regards to it being supported on E-ATX cases. Based on my quick research I think if you have an E-ATX case then you are cool with having a SSI-CEB board.

[Ondra]

Is it possible to disable AVX in bios? Just curious... ;)

[Juraj]

Is it possible to disable AVX in bios? Just curious... ;)

I don't think so, I searched everywhere :- ). Neither to se the clocks for it.

[Ryuu]

I tried building a few testing versions of Corona with different (non-)AVX settings, but it is just as Ondra predicted, there are always some AVX instructions used in the compiler runtime library (which is completely out of our control).

From my point of view, there are only two solutions to this problem:

• Switching to a different compiler - This would probably have to be an older version of Visual Studio (2010?), which is something we really don't want to do.
• Taking full advantage of the AVX(2) instruction set - Right now there is no AVX explicitly used in Corona itself and the compiler is apparently unable to take full advantage of this instruction set even with appropriate settings. This means that we would have to code AVX optimizations by hand. It would be relatively easy to do this for code like texture loading or denoising, but the rendering core itself is a whole different beast.

I'm afraid there is very little we can do at this moment. I'm always happy to rewrite any part of the code to optimize it, but this would mean much bigger changes than what we are able to do at this point. I'll definitely keep searching for other solutions.

[Juraj]

It's extremely little issue, I doubt there is a lot of people with Broadwell-E E5 Xeons, and we loose just marginal performance. It was still educating to learn this.

Cool that you had looked into this.

[fobus]

Is there a point to choose 18-cores (or 22-cores) CPU for render node? Will it use all 72 (88) threads in corona in real rendering process (not only in Benchmark application)? Or may be it will be better to use 14- or 16-cores CPUs (render noe PC)?

[Juraj]

For node, you simply want the highest overall performance. The Xeon line-up is structured that there is never a model which has same performance at lower or higher-core count, it goes like sinusoid instead. Currently the highest-performance yielding models are the 20-22 (there is even OEM 24 version) cored ones.

For workstation though, it's worth sacrificing a little bit performance and go for the lesser cored, higher clocked versions for few reasons:

1) Avoid processor groups like plague. (Non-server) Software developers didn't get the memo Microsoft introduced such feature. Corona and Vray work fine though.
2) AVX under-clocking base-clock and turbo bins. AVX instructions further lower the base clock, which Xeons already have rather low, and even the turbo clock is lower in such case, doesn't make big difference for rendering but it could in some other cases.
3)The weird issue with metro start-up in Windows 10. I honestly have no idea what to do with this, but I didn't had much time to investigate yet. It's ok for node, but very problematic for workstation>
4)You want the highest turbo bins. Currently most of the line-up has pretty flexible and powerful turbo bins, so even the low-clocked Xeons can go from 2.6 into 3.5Ghz for single-threaded processes. These makes v4 Xeons excellent workstation CPUs. Certain ES/OEM versions don't have this ;- ) So unless you're buying "authentic" or latest QS release of CPU, don't use them as workstation.
 

Because of this, it almost doesn't matter what kind of Xeon clock you get, because the turbo is so flexible (this wasn't the case in previous generations). Higher clocked Xeons in v4 don't make better workstation, pretty much all the line-up works the same, although some go up to 3.8+ Ghz in single-core turbo bin. It's because of the above issues you want to avoid it for WS.

[fobus]

So it will not be a problem to use all 88 logical processors (in dual 22-core system) with Corona? No need to setup any sort of DR to use all threads or something else?

About AVX. Is it applicable only to Xeons v4 or every Intel CPU? I tested Corona benchmark with 2600k and 3930k and hasn't seen any sort of lowering frequency. Turbo Boost was used to rise frequency to max for all cores and I had no issues with it.

[Ryuu]

Will it use all 72 (88) threads in corona in real rendering process (not only in Benchmark application)?

Corona benchmark uses the same code that is used for the actual rendering process (albeit slightly older because we don't update the benchmark that often).

I just tested this on 6900k and I'm getting 3.5 GHz in both Corona 1.3 benchmark and Cinebench R15. The CPU is rated at 3.2 GHz base frequency and 4 GHz single core turbo boost. This seems quite fine to me, nothing unexpected here.

[fobus]

6900k has only 16 threads so it will be impossible to see more than one processor group in windows. So Turbo Boost is working as expected. My question is will Corona use more than one processor group if threads more than 64 present.

[Ondra]

6900k has only 16 threads so it will be impossible to see more than one processor group in windows. So Turbo Boost is working as expected. My question is will Corona use more than one processor group if threads more than 64 present.

I feel that we need to make a clear statement because there is some FUD regarding Corona and processor groups.

YES, Corona can handle any number of threads in the system with no problems and with 100% efficiency

We tested it multiple times, and we have reports from multiple people that it works fine. You can even easily artificially create multiple processor groups on your system with less than 64 threads and see for yourself, if Corona or any other software can handle it (https://msdn.microsoft.com/en-us/library/windows/hardware/ff542298(v=vs.85).aspx), so go ahead if you have any doubts :D. There are apparently a shitton of problems when using processor groups, but Corona is not one of them ;)

[Ryuu]

6900k has only 16 threads so it will be impossible to see more than one processor group in windows. So Turbo Boost is working as expected. My question is will Corona use more than one processor group if threads more than 64 present.

Sorry, I though we were still on the topic of Broadwell clock frequencies when AVX is used.

Besides what Ondra said, you can also see the benchmark results for yourself: https://corona-renderer.com/benchmark

Windows by default assigns logical cores from one physical CPU to the same processor group, so a 2x22 core computer would end up with two groups of 44 threads each. If Corona was not properly managing threads across different processor groups, this computer would have the same result as 22 core computers, which is not the case.

[Juraj]

I wrote that Corona (and also Vray from 3.3 service pack) works 100perc. with processor groups :- ).

Regarding AVX, this isn't Haswell-E/Broadwell-E  "issue", but Haswell-E/Broadwell-E Xeon E5 v3/v4 thing, apparently because of their tighter thermal design (see the picture I attached from Intel presentation). It's about 300Mhz difference in turbo-bin depending on individual chip, I couldn't find where Intel actually lists this info.
But it doesn't do any kind of havoc, this is just one of those 'potential' losses, these CPUS are still fast as f*** :- ). Like....really nice.

[Ondra]

I wrote that Corona (and also Vray from 3.3 service pack) works 100perc. with processor groups :- ).

I also wrote it many times, maru did, ryuu did, yet people still have doubts ;)

[fobus]

Regarding AVX, this isn't Haswell-E/Broadwell-E  "issue", but Haswell-E/Broadwell-E Xeon E5 v3/v4 thing, apparently because of their tighter thermal design (see the picture I attached from Intel presentation). It's about 300Mhz difference in turbo-bin depending on individual chip, I couldn't find where Intel actually lists this info.
But it doesn't do any kind of havoc, this is just one of those 'potential' losses, these CPUS are still fast as f*** :- ). Like....really nice.

You mean that Turbo Boost can't boost from nominal base frequency or it will lower уму base frequency down? If I have 2699 v4 with 2.2GHz (2.8 all cores with TurboBoost) I will end up with 1.9GHz lowered CPU in Corona? Or it will boost it to 2.5 only (2.8GHz -300MHz)?

I wrote that Corona (and also Vray from 3.3 service pack) works 100perc. with processor groups :- ).

I also wrote it many times, maru did, ryuu did, yet people still have doubts ;)

Sorry... I got it now. Great, but not for AVX.

[alexyork]

Conclusion seems to be that these latest 18-core+ processors offer the expected rendering power and play perfectly nicely with Corona, but have major issues with Windows relating to Processor Groups, causing error messages, system slow-downs and requiring regular reboots, which is fairly crippling for workstation use.

So we're probably going to go with the latest v4 16-core (2697A-V4) to hopefully avoid this. We'll of course report here with results next week and will submit benchmarks for comparison. In theory they should offer basically identical performance to the 18-core 2697-V4.

Of course now with Brexit the pound/dollar is complete rubbish and the price of these things is through the sky...

[Juraj]

Regarding AVX, this isn't Haswell-E/Broadwell-E  "issue", but Haswell-E/Broadwell-E Xeon E5 v3/v4 thing, apparently because of their tighter thermal design (see the picture I attached from Intel presentation). It's about 300Mhz difference in turbo-bin depending on individual chip, I couldn't find where Intel actually lists this info.
But it doesn't do any kind of havoc, this is just one of those 'potential' losses, these CPUS are still fast as f*** :- ). Like....really nice.

You mean that Turbo Boost can't boost from nominal base frequency or it will lower уму base frequency down? If I have 2699 v4 with 2.2GHz (2.8 all cores with TurboBoost) I will end up with 1.9GHz lowered CPU in Corona? Or it will boost it to 2.5 only (2.8GHz -300MHz)?

Both (base-clock and turbo) are lowered. But because turbo kicks all the time (I consider turbo to be the "true" baseclock since you can't overclock them, well you "can" ...but, that's different topic)



But how much the pictured difference is depends on individual models, I can't find where Intel writes this. This is only for code that uses AVX.


Btw, Windows 10 doesn't require reboots. The issue is with single process that leaks after start menu usage, and simply needs to be restarted in task manager. That is still super annoying though. And processor groups sucks for non-aware apps ( Unreal 4 light baking sees only one....).

[fobus]

We've bought 20 Cores XEONs - all cores areused in rendering. It's good.
AVX is in use and lowering cores speed - it's bad.

[Ryuu]

We've bought 20 Cores XEONs - all cores areused in rendering. It's good.
AVX is in use and lowering cores speed - it's bad.

Engineering samples usually have lower base & turbo frequencies and the frequency scaling may be even more drastic than what is seen in the production version.

[fobus]

Yea. But it's still quite fast. Let's hope that we'll see AVX gone or will be used to increase speed instead of decreasing it.

[Juraj]

You know it's funny, because in actual rendering performance, it's fast like it didn't even underclock :- ).

So I wonder how much faster it would be otherwise... (maybe not at all, maybe the AVX does speed-up something, although all my friends working with it assured me it's minimal )

[fobus]

So I wonder how much faster it would be otherwise... (maybe not at all, maybe the AVX does speed-up something, although all my friends working with it assured me it's minimal )

May be. But I would like to seу maximum clock speed as it can be without AVX.

[Juraj]

Yeah, that's the biggest crap "feature" ever. Thermal design my ass..

[peterguthrie]

In case anyone was wondering, the recent Windows 10 anniversary update thing FIXES the start menu and the runtime broker issues we were having! woohoo!

(still cant set affinity though)

so microsoft were obviously aware of the issues, they just didnt make people aware of them

[Fluss]

Just received my dual 2696 v4 OEM. Max all cores turbo @2.8GHz.

Run @2.8GHz all cores in absolutely any app (Vray, cinebench, mining etc etc...), except in corona -> all cores @2.6GHz. So are you really sure this is not a corona related issue?

[FrostKiwi]

So are you really sure this is not a corona related issue?

No commercial renderer before was based off of Embree from the ground up. This is the first time Instruction sets have been used this extensively and optimized, by the very people who build the processors themselves. Even Vray does not fully integrate it.
This collides with this stupid TDP limit "feature" however, as we near a proper 100% full utilization.
It's actually quite goddamn impressive what modern Instruction sets can do.
So it really is not corona's fault. (Maybe for being this fast...)

[Fluss]

OK I understand better. I've just noticed Arnold is running @lowered turbo too.

[Vuk]

Just tried Corona benchmark with 2698v4 es Xeon cpu. I'am guessing this is AVX issues since in Cinebench all cores are running on 2.3 ghz while in Corona benchmark they run on 1.9 ghz? Also tried in distribute rendering in a standard scene not Corona benchmark same problem. Is this issue fixable? Motherboard is Asus z10pe-d8 ws. Haven't tried to play with bios.

[Juraj]

According to recent posts from devs, not really since Intel injects AVX instruction automatically apparently.

Nonetheless if you do some scaling comparison you'll see it's not actual (negative) performance hit. So it's just Intel's decision to keep low turbos for their server cpus (non-WS xeons) to avoid thermal issues.
So it's unclear if ignoring AVX would actually bring faster rendering in Corona. Would be good to test it somehow but not possible it seems.

[Vuk]

Just tested 2 other Xeons that are oem and qs. Oem works exactly at same frequency in both CB and Corona benchmark. QS is overclocked about 100 mhz and only shows in CB while in Corona is 100mhz less. Apparently the biggest hit is taken by ES cpu's. Don't know how its not a negative thing since it is 400mhz less per core on 20 cores so its also slower rendering time. Thnx for the answer Juraj but I dindn't really understand if this can be bypassed in bios or not?

[Juraj]

Were the QS/OEM also v4 generation ?

The one thing that ES has trouble is that it doesn't have the adaptive AVX turbo that QS/OEM/Retail has, which is under-clocking only the cores that contain AVX instruction. ES will always underclock the most.

It can't be adjusted in bios, the only bypass is by Corona to restrict the AVX. But Ondra said recently he doesn't know how.

[Vuk]

Thnx for the quick answer Juraj. No qs is v3 while oem is v2. Seems that vray ignores AVX instrucions and works at same clock speeds as CB.

[Juraj]

Vray also isn't based fully on embree, although I am not sure if the Embree support isn't on by default there too.

It's only V3/V4 that have the AVX-Turbo concept and mainly V4 where it's most prominent.

By perfomance comparison I meant that I compared the ratios of my v2 and v4 xeons in Cinebench and CoronaBench. The fact that my 2698v4 ES underclocked didn't skew this ratio negatively, which means that the AVX instruction did potentially offset performance in positive way.
This is actually the argument that Intel keeps making. That despite lower clock while using AVX instruction, the actual net performance is higher.

This doesn't take into consideration that ES have worse turbos in general so the net performance is probably worse, but not as worse as pure frequency difference would hint at. The end takeaway is that frequency simply isn't everything when it comes to all the modern instruction sets.

[Zray]

I have 2 computers: dual 2683v3 and Giga X99 + 2696v3. The dual xeon computer seems that it doesn't have any problem with turbo boost speed (2.5 when rendering - full load). However, the 2696v3 turbo boost is just 2.6 (actually, it should be 2.8 - same issue with some other people in this topic). That's really annoying and I suppose the rendering is a little bit slower.

[Rimas]

I know this is a wee bit off-topic, but I'd much rather prefer to work with top-end i7 chips than Xeons... Had a 20-core Xeon machine - it was slow in single-threaded tasks (say....MAX viewport!) at 2.6GHz and got superseded by a heavily overclocked 8-core i7-5960X (from 3.2GHz all-core to 4.5GHz all-core, on water). cheaper, much more responsive and a pleasure to extract more power from by overclocking. You do lose out on being able to have hundreds of gigs of RAM and having more than one socket, but for a workstation I wouldn't ever buy a Xeon anymore...at least not with those pathetic clocks...

[Juraj]

I know this is a wee bit off-topic, but I'd much rather prefer to work with top-end i7 chips than Xeons... Had a 20-core Xeon machine - it was slow in single-threaded tasks (say....MAX viewport!) at 2.6GHz and got superseded by a heavily overclocked 8-core i7-5960X (from 3.2GHz all-core to 4.5GHz all-core, on water). cheaper, much more responsive and a pleasure to extract more power from by overclocking. You do lose out on being able to have hundreds of gigs of RAM and having more than one socket, but for a workstation I wouldn't ever buy a Xeon anymore...at least not with those pathetic clocks...

Latest gens of Xeons can overclock single/two/four cores up to 3.6+ Ghz, they don't suffer from slow single-thread tasks at all.

And previously, i7s were much slower in multithreaded, making them poor choice for people who do a lot of quick test renders and interactive. Having monster machine can really speed up look development workflow in cpu renderers.

But right now I would agree. The best workstation right now is overclocked i9 7980XE. Basically Xeon machine.

[Zray]

I know this is a wee bit off-topic, but I'd much rather prefer to work with top-end i7 chips than Xeons... Had a 20-core Xeon machine - it was slow in single-threaded tasks (say....MAX viewport!) at 2.6GHz and got superseded by a heavily overclocked 8-core i7-5960X (from 3.2GHz all-core to 4.5GHz all-core, on water). cheaper, much more responsive and a pleasure to extract more power from by overclocking. You do lose out on being able to have hundreds of gigs of RAM and having more than one socket, but for a workstation I wouldn't ever buy a Xeon anymore...at least not with those pathetic clocks...

Probably you're right, especially with your xeon machine (I guess it is a quite old one). Since xeon E5 v3 and v4 announced, it's much better than E5 v1.

[peterguthrie]

Do you know when the new gold xeons are going to be available Juraj or anyone?

[Rimas]

I know this is a wee bit off-topic, but I'd much rather prefer to work with top-end i7 chips than Xeons... Had a 20-core Xeon machine - it was slow in single-threaded tasks (say....MAX viewport!) at 2.6GHz and got superseded by a heavily overclocked 8-core i7-5960X (from 3.2GHz all-core to 4.5GHz all-core, on water). cheaper, much more responsive and a pleasure to extract more power from by overclocking. You do lose out on being able to have hundreds of gigs of RAM and having more than one socket, but for a workstation I wouldn't ever buy a Xeon anymore...at least not with those pathetic clocks...

Latest gens of Xeons can overclock single/two/four cores up to 3.6+ Ghz, they don't suffer from slow single-thread tasks at all.

And previously, i7s were much slower in multithreaded, making them poor choice for people who do a lot of quick test renders and interactive. Having monster machine can really speed up look development workflow in cpu renderers.

But right now I would agree. The best workstation right now is overclocked i9 7980XE. Basically Xeon machine.

3.6 sounds slow to me, considering my grandpa Haswell-E chip can handle 4.6-4.7 at the same core amount, albeit the IPC is slower than the new chips. I think I saw that i9 handle 4.8GHz on all cores somewhere, which is huge IMO.

[Nejc Kilar]

@Rimas,

While I agree with what you are saying I only agree to an extent :P I have a dual Xeon that reaches 3.8ghz using a single core and its slower in lightly threaded tasks than my i7 4790k @ 4.4ghz. From my experience the difference is not that huge. Especially not so in the viewport. Some sims take longer to run but generally for most stuff I don't notice the difference.

That said, in IPR and any multi-threaded tasks (rendering, video conversion) the difference is obviously huge.

So from my point of view its like taking a little out of the single threaded speed and gigantically improve multi threaded speed.

Ultimately I do think it depends on what you do. If you mainly do smaller / medium sized scenes the difference I think its negligable compared to the rendering boost (IPR).

If you do VFX and mainly render / sim on 2-4 threads, ha, its probably better to go the other route and invest into a 4.8 or whatever CPU :) As a matter of fact, the 4790k I used to use before the Xeons is now the perfect workstation for my significant other. She is using it in Marvelous Designer and the speed gains on those 2-4 threads are really big for her.

Being primarily a c4d user myself I can also say that I haven't noticed a big difference in viewport speed in the said app. C4D traditionally really likes fast cores.

Even if the viewport has 20fps instead of 30fps (fictional numbers, haven't done the tests) I can still get the previews so much faster than before because of the IPR speed... Obviously something like MD would be a different story probably.

Now like you said, and I totally agree with you on this part, if the max speed for your CPU is like 3.2ghz I think you'll definitely notice it being slower in quite a few areas. Heck, the speeds you posted a few posts back are lower than my current all-core boost.

The biggest drawback is obviously that to get above 3.4 ghz on a dual Xeon system you probably need to pay through the nose for it.

I mean thats just my point of view anyway. Like I said, I think it depends on what you are doing. From that point of view  GPU rendering seems fun because you can load it on a highly clocked / low core CPU and still get great rendering speeds. There are obviously other limitations there though...

I was very skeptical of the single threaded speed before buying the Xeons but in the end it turned out to be a great choice for what -I DO-. :)

[Juraj]

There is a point where placebo effect becomes a fetish ;- ). Personally, below 5 GHz, even sending files to trash-bin feels slow.

[Rimas]

I haven't tested with the latest 3DS MAX version, but the differences between my old Xeons and the overclocked 5960X were truly big (like 50% or more) in viewport framerates. Cause - MAX viewport is single-threaded. I had a bit of a battle over it with Autodesk's technician once as it seemed pathetic to me. I noticed it when I upgraded my GPUs and a scene that rendered at 15fps never improved speed, which prompted a "WTF???" from me :D

Anyway, I do a lot of Photoshop, Illustrator, Muse and Lightroom work eith a decent bit of 3D modeling and rendering, so I primarily favor fast cores for snappyness in those programs - I hate it when software is slower than me :D (Hell, I work on a 165Hz display too xD)
For rendering - my 5960X still does very well and we get by. Sure would be nice to tap into the lastest i9 CPUs or dual Xeons over 64 cores, but there's no need for it now as I'll just backburner or DR the bigger jobs overnight to the other workstations.
I also have an identical system at home with a bunch of GTX  1080s and 1080Tis which I use for FStorm rendering and gaming - the 5960X is also very well-versed there. But I appreciate it that my needs are different - I like a snappy all-rounder of a machine. :)

[SaY]

Do you know when the new gold xeons are going to be available Juraj or anyone?

They are available for a while now. At least Platinums, running a pair of them in my main rig. No overclocking though,  but ASUS may allow some with their C621e Sage m/b - not available yet.

[hrvojezg00]

Do you know when the new gold xeons are going to be available Juraj or anyone?

They are available for a while now. At least Platinums, running a pair of them in my main rig. No overclocking though,  but ASUS may allow some with their C621e Sage m/b - not available yet.

I`m building 3 Xeon Platinum systems atm, waiting for memory as none of our current systems memory don`t work with it. One has C621e mobo, will post oc-ing resaults if any.

[peterguthrie]

why platinums? quad cpu??

[hrvojezg00]

No, just found great bargain on 28 core ES platinums on ebay. All 2cpu systems.

[SaY]

I`m building 3 Xeon Platinum systems atm, waiting for memory as none of our current systems memory don`t work with it. One has C621e mobo, will post oc-ing resaults if any.

Cant wait to hear the o/c results.
Where did you get the C621e mobo? I can't find it anywhere..

[hrvojezg00]

I`m building 3 Xeon Platinum systems atm, waiting for memory as none of our current systems memory don`t work with it. One has C621e mobo, will post oc-ing resaults if any.

Cant wait to hear the o/c results.
Where did you get the C621e mobo? I can't find it anywhere..

Got it from my local supplier (Croatia), I think it came from Germany.

[Juraj]

SaY, how many PCs do you have ? And do you build these for fun or for your office ?

To be honest I am jelly of these plat xeons ES. Had no idea people were already buying these, nor that they were available.

[SaY]

Juraj, I tried to PM you but it seems to be blocked - can you PM me?

[Juraj]

I have the system disabled to avoid having messages all over the internet :- )

I have email juraj(dot)talcik@yahoo(dot)com or skype Juraj_Talcik

[peterguthrie]

PM me too!

[hrvojezg00]

SaY, how many PCs do you have ? And do you build these for fun or for your office ?

To be honest I am jelly of these plat xeons ES. Had no idea people were already buying these, nor that they were available.

For business only of course. There are 4 of us in the office, all on 2cpu e5 xeon workstations (a xeon platinum workstation is being built atm), and have a render slave rack which is constantly being updated. Right now its one e5 2699v4 dual, one quad e7 8890v3 and waiting for memory for two xeon platinums to fit in. Everything is setup on 10gbe network to make sure all render nodes are starting up fast when needed, I use them all even for quick drafts. Works great!

edit: All cpus are ES!

[Mor4us]

maybe some of you are interested too :D