Author Topic: 10 Gigabit Ethernet Performance  (Read 9932 times)

2018-08-27, 11:33:42
Reply #15

hrvojezg00

  • Active Users
  • **
  • Posts: 270
    • View Profile
    • www.as-soba.com
I've been running 10GBE setup for 2 years now. It's drastic improvement, and imho absolute must for Distributed.

With 1GB, distributed would regularly completely stop because all those large .exr files being sent are simply too big. This happens if you have more then 1 node and render something like 8k renders.
Making the intervals too long or amount of pixels being send lower invalidates the usability of distributed so I never opted for this.

DR is still too slow to start, I don't know why this is. The scenes on slaves always take like 10 times the time during the "pre-pass" stages than they do on workstation. I always check task manager and see that the damn slave is using 1-core for like 2 minutes before it moves to other part of prepass before finally starting to render.
I basically gave up on using DR for test renders, it's uselessly slow so I only use it for finals. I don't have 5 minutes before it starts sending passes, which is what it usually takes.

In my experience, parsing on slaves depends heavily on total (archive) size of the file being rendered. Alot of 8k textures and so take time to read. Also, after first render of the scene, every other starts sooner due to downloaded assets on each slave. Personally, I use all slaves on all drafts/finals, rarely takes more then 2 min for all slaves to start rendering. DDR3 memory fills up alot slower then DDR4, Juraj how many slaves with DDR3 you have?

2018-08-27, 11:51:25
Reply #16

Juraj

  • Moderator
  • Active Users
  • ***
  • Posts: 4743
    • View Profile
    • studio website
All my slaves are DDR4 (2400Mhz CL13 ECC), SSD, 2698v4.

Yeah, small scenes can take 2 minutes, but even that is too long. The scene transfer takes <1 second, the 3dsMax is already opened, for some reason "downloadding assets" happens rather long regardless of instant access to 10gbe fileserver, this one is mystery to me as well, and the the endless one thread parsing happens.
The whole is much faster if I just open the scene and hit render. So something is much slower if it's done externally by DR. I am most buffled by the long period of one-thread usage.

(( Why is it downloading assets at all ? Does it mean "Reading assets" ? Because I don't have anything stored locally, everything is fileserver linked))

So the fastest are 2 minutes, but the average before first pass comes is 4-5 minutes. Yes my scenes are rather big but that just highlights how inefficient the process is.

In 4-5 minutes I want to have fully rendered preview not wait for passes to start coming. I'll see how much the 2990WX + 1080ti/2080ti will help in this process.


PS: Hrvoje, absolutely kickass NYC project  (Kent) !! Top grade
« Last Edit: 2018-08-27, 11:55:52 by Juraj Talcik »
Please follow my new Instagram for latest projects, tips&tricks, short video tutorials and free models
Behance  Probably best updated portfolio of my work
lysfaere.com Please check the new stuff!

2018-08-27, 12:56:20
Reply #17

hrvojezg00

  • Active Users
  • **
  • Posts: 270
    • View Profile
    • www.as-soba.com
All my slaves are DDR4 (2400Mhz CL13 ECC), SSD, 2698v4.

Yeah, small scenes can take 2 minutes, but even that is too long. The scene transfer takes <1 second, the 3dsMax is already opened, for some reason "downloadding assets" happens rather long regardless of instant access to 10gbe fileserver, this one is mystery to me as well, and the the endless one thread parsing happens.
The whole is much faster if I just open the scene and hit render. So something is much slower if it's done externally by DR. I am most buffled by the long period of one-thread usage.

(( Why is it downloading assets at all ? Does it mean "Reading assets" ? Because I don't have anything stored locally, everything is fileserver linked))

So the fastest are 2 minutes, but the average before first pass comes is 4-5 minutes. Yes my scenes are rather big but that just highlights how inefficient the process is.

In 4-5 minutes I want to have fully rendered preview not wait for passes to start coming. I'll see how much the 2990WX + 1080ti/2080ti will help in this process.



Agreed, hope dev team will work it out! Corona is very power hungry, so flawless DR is a must!

PS: Hrvoje, absolutely kickass NYC project  (Kent) !! Top grade
Thanks, well appriciated!

2021-05-04, 12:29:24
Reply #18

EmerSharif

  • Users
  • *
  • Posts: 3
    • View Profile
Hi guys,

sorry for waking up this thread!
Like you i was tired of experiencing slow 3dsmax open and saving huge scene, slow "downloading assets" on DR servers, etc.

Last week i tried a 10gb network with SFP+ cables not ethernet.

On our network via windows all is extremly fast, the 10gb bandwith is fully used wille transfering data (we have working assets and library on fast NVMe drives on the server.

BUT, in 3dsmax (2020) it's like hell!!! Every action who need network access is slow as i never seen before (refreshing thumbnail on material library), loading scene, etc.... It's throtelling on few Mbits transfert??!

I tried few differents settings on networks cards and switch like "jumbo frame" on multiple sizes.
Nothing help!

I'm really tired of 3dsmax sometimes.................

Any ideas?

2021-05-04, 13:47:51
Reply #19

alexyork

  • Active Users
  • **
  • Posts: 699
  • Partner at Recent Spaces
    • View Profile
    • RECENT SPACES
We also tried 10Gbe but found little to no real-world improvement with 3ds max reading/writing. Or even Photoshop for that matter. Some read performance benefits here and there but nothing anywhere near worthwhile compared to the cost. As you say, max itself appears to be the main bottleneck. For stuff like DR or video editing that relies heavily on file transfer it might be well worth it.

But maybe others have had more success lately with this?
Alex York
Partner
RECENT SPACES
recentspaces.com

2021-05-04, 14:09:35
Reply #20

Dalton Watts

  • Active Users
  • **
  • Posts: 210
    • View Profile
I've given up DR since 2016, that's when I bought my dual Xeon 2699V3 workstation. But I only tried DR through 1gbe and generally it took too long for slave(s) to kick in. From those with 10Gbe setups how much time does a slave take to actively participate in rendering?

I've also given up on centralizing every file on a NAS (on 1Gbe). It was paaaaiiiinfully slow for Max to read.

So, my next best bet would be to hook up a 5950X workstation to my dual Xeon 2699V3 through 10Gbe with files being stored on the main Workstation, but I'm unsure how much time would take for the slave to participate.



2021-05-04, 15:20:36
Reply #21

EmerSharif

  • Users
  • *
  • Posts: 3
    • View Profile
We also tried 10Gbe but found little to no real-world improvement with 3ds max reading/writing. Or even Photoshop for that matter. Some read performance benefits here and there but nothing anywhere near worthwhile compared to the cost. As you say, max itself appears to be the main bottleneck. For stuff like DR or video editing that relies heavily on file transfer it might be well worth it.

But maybe others have had more success lately with this?

Hi Alex,

exacltly same feeling here, muche money for... almost nothing! 3Dsmax an d network access with huge scene and assets, just hell.
Technicaly, the best we tried here, is to agregate mutiple gigabit ethernet ports (4 x 1Gbits) to make a unique 4Gbits. It's pretty nice working, and easy to setup under Windows server 2019. It's help for the server to send data on render slaves during computing on DR or animation.

I'm going to send back all our 10Gbits hardware.


Best,

M.

2021-05-05, 19:05:07
Reply #22

Vuk

  • Active Users
  • **
  • Posts: 113
    • View Profile
I have the same experience. I am using a Raid 0 NVME Storage pool consisting of 4x2TB 970 Evo Plus's just for the PROJECTS pool with all the WS's running 10gbe Nic's and 10gbe cabling throughout the whole office.
The 10gbe speeds are completely saturated during simple copy/paste file transfers yet in real-world use, at least for "US", there is absolutely no difference.

Now I could sit here and write 5 other pages on how I tried several configurations from running Raid 5 and 6 on standard HDD's to running single ssd's and finally running a raid with Nvme m2 ssd's but this should be as simple as possible. I assume this would only be interesting to a few "geeks" of us here and most people wouldn't understand and I don't blame them. I myself lost a lot of time to learn this and in the end, it turned out to be a complete disappointment in both performance and my wallet...

I tried every possible test that suits our workflow. Scene loading times vs 1gbe, Render node loading times vs 1gbe, Photoshop, you name it :). As I wrote before the most important thing is always the CPU and it comes ahead of everything (network, ssd, ram) the faster the cpu the faster the loading times will be :).
Just by changing our old xeons from the farm with newer threadrippers we have seen a massive gain in performance. Those machines work on a much higher boost clock than the Intel Xeons and load heavy scenes ( we had files of around 3-4gb in size ) much faster. A quick real-world example would be that we used a Dual Xeon Platinum 2x24 core machine as a Backbruner Server with DR rendering on. This machine vs the 3970x took sometimes even up to 15 minutes just to load a scene and start rendering while the 3970x did the same scene loading in 5 minutes. We had to deliver 18 images that day in 5k resolution. 18 images X 10 minutes more time to load on a Xeon machine is 180 minutes more spent on just loading the same scene 18 times over on a different machine. In my book 180 minutes more is a lot of time lost in an 8-hour working day.

I haven't compared exporting the passes at the render end and saving those big CXR files. But at the end of the day how many times are you doing that during your whole working day? It's not even a tiny 1% of your time so who cares if it's a few seconds more right?

I burned a lot of cash on this 10gbe venture and to all people who plan on doing the same and still using the same software (99% of us use on this forum -3ds Max, Corona, Photoshop), mainly for 3d visualization purpose I say, save your money and invest in a high-end workstation or a rendering node instead of wasting it on something you won't get any real benefit.

Now I suppose Juraj will write (as he did before :P) that I am completely in the wrong with this one and that he has some amazing performance gains. But from what I see I am not the only one complaining about this and I have no real reason to lie either :). On the other hand, if you do video editing then this is definitely the route to go since those tasks mainly use sequential read and write operations that fully benefit from 10gbe speeds.




« Last Edit: 2021-05-06, 12:44:40 by Vuk »

2021-05-31, 17:31:57
Reply #23

arqrenderz

  • Active Users
  • **
  • Posts: 990
  • https://www.behance.net/Arqrenderz1
    • View Profile
    • arqrenderz
HI Vuk, thx for all the details, in my case scene parsing times got 10 to 15 seconds faster on 10gb vs 1gb network. I think its a good thing to have on a server that has to share its files with multiple users, if the server has just 1gb network it will saturate in a second, one thing that worked was 3dsmax saving and loading times on a saturated network.

2021-06-09, 19:07:13
Reply #24

Juraj

  • Moderator
  • Active Users
  • ***
  • Posts: 4743
    • View Profile
    • studio website
10G network will not increase any performance anywhere unless there was previously a bottleneck.

Since for reading/writing assets from 3dsMax/Photoshop/etc.. the main bottleneck are those softwares, you won't see much faster loading/saving.
For distributed rendering alone, 10G is massive benefit. Streaming 8K renderings using some medium fraction of megapixels every 60 seconds, even single node can reach the limits, let alone multiple.

Network is like memory, unless you ran against the wall previously, increasing it will not do anything, there is nothing magical about it.
Please follow my new Instagram for latest projects, tips&tricks, short video tutorials and free models
Behance  Probably best updated portfolio of my work
lysfaere.com Please check the new stuff!