What to expect from distributed rendering 1 frame with 4 to 8 render nodes?

[PixelPioneer]

Hi everyone!

We have stepped away from another render engine (you know who you are ;-) and we're new to Corona. LOVING IT!

We are working on massive scenes that render too slow in the interactive renderer, even after heavy optimization.

So we are thinking about distributed rendering with 3 to 7 slave nodes besides the master node.

Two questions for those with experience with these kinds of setups:

1) How linear is the reduction in render time with each additional render slave? In other words, 1 master + 1 slave = ~50% of master-only render time? 1 master + 7 slaves = ~12.5% of master-only render time?

2) Can we set the synchronization interval to 1 second to actually get a quick and valuable update (after all slaves have preloaded all data and are ready to rock), or doesn't the tech work that way due to network delay et cetera?

It would be great to understand what to expect before we make such a big investment.

Thank you so much for your feedback in advance!
Nando

[TomG]

As a note, DR doesn't work with IR, in case that was what you were thinking of specifically. It only comes into play when doing full renders.

Doing an interval like 1s would likely cause your network to collapse, as too much data pinging back and forth :) For final renders, there's no hard and fast calculation, but in general if all nodes are equal, I'd say using two would reduce render time to maybe 60% rather than a full 50% to allow for data being moved back and forth and some things like "extra passes" being done depending on when the machines check in with each other. But it will depend on the scene, network speeds, etc. - should be a good enough ballpark though to decide whether it is worth it.

[PixelPioneer]

Thank you Tom, much appreciated!

Indeed I already read that DR doesn't work with the interactive renderer, that is understandable.

Then as a final question: would this be the best approach to increase our scene iteration speed, or would you maybe recommend other options like investing in a very heavy machine with maybe multiple processors?

Thanks!

[TomG]

For faster IR, it really does come down to the CPU (well, assuming you have plenty of RAM and are not running low, in which case, more RAM is the starting point!). You can compare how CPUs currently perform with Corona over at https://benchmark.chaos.com/corona/v10/scores?index=1&order=desc&by=rays&my-scores-only=false and can run the benchmark on your current machine, to see how much of an improvement another CPU might get you.

Note that at present in the Benchmark some high core/thread count machines underperform (those with 64/128 or higher) in both the Benchmark and Corona - this does mean the Benchmark results show you what you will get today, but a fix coming in Corona 12 that will see them gain some performance. Also note that there is a separate bottleneck for high core/thread count machines with dual CPUs; again, the Benchmark results will show you what you get today, and the previously mentioned fix should help some, BUT we're still diagnosing the extra issue that comes from the multiple CPUs and we have no timeframe on when we'll overcome that bottleneck. Until we isolate the cause, we can't say how long it will take to fix :( Once we have both issues resolved, we'll release a new Benchmark. For now though the Benchmark is showing the actual performance obtained in Corona 10 / 11.

Those are things to factor in if you are considering buying a very high end machine like some dual Epyc 128/256 monster! What machines are you running on today, out of interest?

[PixelPioneer]

Thanks again, so much valuable info! :)

Our current machines are more targeted towards GPU renderers with RTX4090 cards, i7-12700KF and 128GB RAM, so not optimal for Corona.

Although I have to say that I didn't expect the i7 to perform interactive rendering that well in comparison with a GPU renderer, in fact I'm still somewhat shocked that we're now heading in a CPU direction, not GPU :)

Okay, so assuming that you will eventually fix the dual-CPU bottleneck, do you think the AMD Ryzen Threadripper PRO 7995WX 96-Cores (currently @ ~42M rays/s) would at that point be slower than 2x AMD EPYC 9554 64-Core Processor (currently @ ~32M rays/s)?

And finally, how much RAM is enough for such a setup, because I'm seeing 128GB, 384, 512...? Does it depend on the size of the scene or is there a max?

Thank you for your time Tom!

[Nejc Kilar]

...

Okay, so assuming that you will eventually fix the dual-CPU bottleneck, do you think the AMD Ryzen Threadripper PRO 7995WX 96-Cores (currently @ ~42M rays/s) would at that point be slower than 2x AMD EPYC 9554 64-Core Processor (currently @ ~32M rays/s)?

And finally, how much RAM is enough for such a setup, because I'm seeing 128GB, 384, 512...? Does it depend on the size of the scene or is there a max?

Thank you for your time Tom!

That comparison is a bit difficult to make because 2S systems do have some drawbacks but broadly speaking I would expect the two Epycs to be faster as they are clocked faster (all core) and have more combined cores. Plus you are comparing Zen 4 to Zen 4 so same uArch. As Tom mentioned though it does depend on the bottleneck, how it is fixed etc etc. We'll keep everyone updated on that.

As for the RAM itself, I can only speak from personal experience but I typically buy about as much as I need - regardless of the CPU in use. For me scenes never spill over the 128GB I have so that is the magic number for me. I do have a dual Epyc with 256GB but that is because it was rather impossible to get a hold of 16x 8GB sticks :)

The biggest thing to watch out with memory is that you always have enough to fit your scenes in. If your scene doesn't fit into your memory your OS will start paging the overflow to your SSD / HDD and even though SSDs (NVMEs) are super fast these days they are still a crap ton slower than your system RAM.

There's been talk about having enough memory for high thread count systems so that each core has more to work with but frankly I've never seen any articles or provable points confirming that for when it comes to rendering.

[TomG]

Yes, there is more nuance to GPU than CPU than many would have you believe, and it's not as simple as "GPU is always 100,000 times faster!" ;)

On RAM, I have no problem on 64Gb, but then I don't do a whole lot of rendering of very complex scenes. There's no such thing as "too much RAM", but, money being what it is, it doesn't make sense to buy more than you need as that gets expensive fast. If money is not an issue though, max out that motherboard lol!

And last, on the single vs. dual CPU set ups, it's impossible at the moment to predict what will happen with the dual CPUs, as it all depends on what turns out to be the cause and how we can fix/workaround that, and just what performance increase that leads to. That does make deciding between those machines tricky for now, unfortunately. Other data to consider is what else the machine is going to be used for (video editing? etc) and how the performance compares on those things, there might be enough of a telling point there to make the decision.

Hope this helps!

[PixelPioneer]

Hi Nejc and Tom, we're loving the wonderful support from you guys! :)

Okay well, I think we will 'gamble' on the dual Epycs setup then with 256GB RAM. They're currently not that much slower than the Threadrippers and faster with some of our other software packages. And who knows they might get a speed bump one day, but noted that there are no assurances.

@Nejc: would you mind sharing the motherboard model you're using for your dual Epycs? Always great to have a trusted example.

In a comparison to Corona's interactive renderer, I really like the viewport-integrated Cycles renderer in Blender. Have you ever tried it? (I have checked this forum on it.)

I'm missing a LOT of features (both in Cycles and Blender) and more importantly I prefer Corona's look.

But it's such a joy to have true render engine interactivity in the viewport! With the quick adaptive denoiser, final pixel rendering... there's nothing quite like it.

Such a deep integration of e.g. Corona in 3DSMax, is that even possible with Chaos and Autodesk being separate entities? One can dream :))

Thanks!

[Nejc Kilar]

Well I'm super glad to hear you found our replies helpful - thank you for the kind words!

The motherboard I am using is the Supermicro H12DSi-NT6 - its a dual socket Zen 3 motherboard (you've got Zen 4 CPUs btw). Do note though that if you are going to place this thing into a desktop style case and have it in your office you'll probably struggle keeping the VRMs from overheating. The VRMs on server motherboards are typically "just enough" and aren't necessarily overbuilt hence they rely on that supreme rack airflow that you typically have in server environments.

I currently have it in a Fractal Design Torrent case with pretty much all the stock fans maxed out and it is hitting 90c - at 100c it starts beeping like crazy because danger :) If your server or office environment isn't AC controlled you're probably going to struggle keeping it cool imho so beware. I'm currently thinking of creating a 3d printed ventilation shaft that would direct airflow directly over the VRMs, hopefully that'll help because during summer this thing will otherwise be working in "limited performance" mode :)

[PixelPioneer]

We have no experience with these types of chips, so thank you for mentioning this! The budget for the setup keeps increasing ;)

Not sure what the final setup will look like, but at least now we know the prerequisites.

Thanks again guys, I'll keep you updated on the outcome! :)