Author Topic: More Team Render Woes  (Read 2506 times)

2019-10-16, 23:57:27

BigAl3D

  • Active Users
  • **
  • Posts: 881
    • View Profile
So for some reason, we started getting this strange error when using Team Render. We were rendering a still frame at 4k, with some Depth of Field effect. Previously, we tried an 8k still and could not get it to work at all. Not sure if it has anything to do with Maximum size of packets settings or not, but I've never fully understood those numbers. Although, after we changed to to Interval to 1 and Packet Size to 30, it seems to work better. I just revisited an old thread where TomG talked about large size images and manually setting those numbers to 20s and 256MB.

(Complete Console text attached)

2019/10/16 17:26:55  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:26:56  [Corona4D]
Frame synchronization failed: Communication Error
2019/10/16 17:26:57  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:26:57  [Corona4D]
Frame synchronization failed: Communication Error
2019/10/16 17:26:59  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:26:59  [Corona4D]
Frame synchronization failed: Communication Error
2019/10/16 17:27:00  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:27:00  [Corona4D]
Frame synchronization failed: Communication Error
2019/10/16 17:27:02  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:27:02  [Corona4D]
Frame synchronization failed: Communication Error
2019/10/16 17:27:03  [Corona4D]
Sending chunk 26/30 to the server
2019/10/16 17:27:04  [Corona4D]
Frame synchronization failed: Communication Error

Then the clients start dropping off the render job. Didn't matter if we launched the render from the Mac Pro or the iMac. We have a couple of Windows 10 clients too, but only render clients.


Full-speed non-debug version
Build timestamp: Sep 26 2019 12:25:41
Version: 4 hotfix 3 (core 4 (Hotfix 3))
Cinema version: R20.059 CINEMA 4D Studio (single-license)

MacOS 10.12.4 (16E195)
Mac Pro (late 2013)

2019-10-17, 01:27:04
Reply #1

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5468
    • View Profile
If you mean and interval of 1, that would be 1 second - which would mean data tries to be sent every second, which would most likely lead to network congestion.

The first thing to try is under the Corona Team Render settings, swap to Arbitrated (experimental), if you haven't already. You can start with default values of 5 for Interval and 64 for packet size. If that doesn't work, I would try the numbers I mentioned below - this sends data much less often (only every 20s rather than every 5 seconds), but in larger chunks when it does send data (256MB rather than 64MB). Sending data too often (low interval) can cause your network to overload and data will be lost. Sending chunks that are too small can cause TR to be pretty pointless, as it will take so long to send the data every x seconds in y chunks that it will be slower than not using TR.

Along with that, useful info is:
How many nodes are you using? (the more nodes, the more likely this is to be network congestion)
What is the speed of your network? (the slower the network, the more likely this is to be network congestion)

Cheers!
Tom Grimes | chaos-corona.com
Product Manager | contact us

2019-10-17, 19:45:56
Reply #2

BigAl3D

  • Active Users
  • **
  • Posts: 881
    • View Profile
Thanks for the reply. Yes that makes a lot of sense as far as not overloading the network. No point in sending tiny chunks every second, when you don't need to. Some info on our LAN. Seems to be fast to my little brain.

10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.153/0.186/0.220/0.023 ms

10 Gbit/s


The nodes can range from one to eight. Some scenes are too slow on certain nodes. So if we do use all eight nodes, it would make sense to increase the time interval AND the size of the chunks, correct?


2019-10-17, 20:11:33
Reply #3

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5468
    • View Profile
Good on the network speed, that's a plus!

That sounds like what I would try, for more nodes, longer interval, but larger chunks. If you still have problems, do let us know (I'd be interested to know if you don't have problems, what settings you settled on, as I am sure there will be some variance based on number of nodes and network speed, but we aren't able to test all combinations here to see what works best, so real world facts and figures would be good to see).
Tom Grimes | chaos-corona.com
Product Manager | contact us

2019-10-21, 16:04:32
Reply #4

HFPatzi

  • Active Users
  • **
  • Posts: 138
    • View Profile
After long time of testing, my go to settings are 50s / 50MB in manual mode. Most of the time 11 Machines are beeing used. Network is 1Gbit/s.

Maybe it helps.

2019-10-23, 20:32:04
Reply #5

BigAl3D

  • Active Users
  • **
  • Posts: 881
    • View Profile
Forgot to revisit this thread. We think at some point we may have reduced the interval down to 1 sec when we had problems. The funny thing is, when we got it to work, the number was increased back to the original 5 sec.

2019-10-23, 21:50:39
Reply #6

HFPatzi

  • Active Users
  • **
  • Posts: 138
    • View Profile
I guess it also depends on the size of the image