Author Topic: Team Render failure, how to troubleshoot  (Read 5989 times)

2019-10-04, 12:36:26

Barendby

  • Active Users
  • **
  • Posts: 123
    • View Profile
What does these comments mean? I used to be able to use TR on all my renders but lately it fails on all renders.
I have no idea what to trouble shoot.
I did upgrade to the latest beta but it was like this on the previous beta as well

2019-10-09, 06:56:40
Reply #1

Barendby

  • Active Users
  • **
  • Posts: 123
    • View Profile
Anybody able to guide me?
I noticed sometimes it works, sometimes not. Really unpredictable.
I am about to start a huge project and could really benefit from TR

2019-10-09, 10:12:49
Reply #2

lollolo

  • Active Users
  • **
  • Posts: 222
    • View Profile
I only had such problems when my main machine crashed or (if I remember correctly) when my render client didn't get the correct corona license.

What settings do you have in the Team Render tab of corona? Maybe to high "Maximum size of packet" or something...

2019-10-23, 20:33:25
Reply #3

BigAl3D

  • Active Users
  • **
  • Posts: 879
    • View Profile
FYI - Knowing more details about your project would help. How large is your render size? Animation? How long?

@Barendby We had a similar issue recently and the issue seemed to be when using several nodes in particular, having the "Client Update Interval" too low (sending data more often), then this seems to overwhelm the network. Go to the Team Render tab in the Corona render settings and change Configuration to Manual. Then you can change those numbers. By default it's 5 seconds and 64 MB. If your frames take a few minutes to render, try 30 sec or more. Otherwise, every node is trying to send 64 MB of data every SECOND back over the network. This can get out of hand quickly.

Let us know if this works. Here's a recent thread I had started on this topic.  https://forum.corona-renderer.com/index.php?board=49.0

2020-01-08, 08:38:28
Reply #4

kuechel

  • Users
  • *
  • Posts: 2
    • View Profile
I have the same problem.
In the client's console panel I get the message: Frame synchronization failed: Communication Error and then the client one by one disconnects.

I wonder if it's not too much traffic in the network because if I render on 3 nodes everything works properly but if I connect more then 3 suddenly all of them disconnect.
My ethernet is 100Mb not Giga so I believe that might be an issue. It would good to know for sure though before upgrading the ethernet cards in all the clients and routers.

2020-01-13, 12:21:23
Reply #5

RickIrvin

  • Users
  • *
  • Posts: 1
    • View Profile
We've had similar issues with nodes cutting out after this error message (Frame synchronisation failed) too. Literally just had 3 of 4 nodes drop out just now, always 1 by 1.

It always seems to occur on still renders only, animations are fine. It also only seems to occur when the TR config is set to 'Automatic'.

I've previously set the config to 'Manual', with update interval at 20s and packet size of 5mb. This prevented cut outs of render nodes, but the render would be no quicker than if I rendered on a single machine.

Reading through some comments and some responses from TomG, it's suggested having a larger packet size for larger resolution renders. So, personally, Im gonna try 100mb at 40-60s intervals and se how I get on. I'm not entirely hopeful, but Ill report back if I find it beneficial.

Rick

2020-01-13, 14:34:55
Reply #6

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5434
    • View Profile
Interesting on animations being fine, but stills being a problem - with animations, each machine usually handles its own frame, and so there isn't the constant interchange of data that there is in a still; so it does all still point to network congestion from too much data being sent, which means the packet size and frequency settings would still be the first place to look. Let us know how your tests go!
Tom Grimes | chaos-corona.com
Product Manager | contact us

2020-01-13, 14:40:25
Reply #7

houska

  • Former Corona Team Member
  • Active Users
  • **
  • Posts: 1512
  • Cestmir Houska
    • View Profile
It always seems to occur on still renders only, animations are fine. It also only seems to occur when the TR config is set to 'Automatic'.

Have you tried the new "Arbitrated" mode?

2020-01-13, 16:17:54
Reply #8

celmar

  • Active Users
  • **
  • Posts: 214
    • View Profile
I'm not sure if it's a valid question:  I thought that, with the monthly fee, I get Corona on my main machine, and 3 other computer on the team render... well, maybe I'm wrong on that... actually, my 2nd machine has Corona5 installed, is on net, but asy "No license loaded", and I don't know how to "give" it the license... if somebody can make my ideas clearer??? thank you!!!

2020-01-13, 16:22:06
Reply #9

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5434
    • View Profile
I'm not sure if it's a valid question:  I thought that, with the monthly fee, I get Corona on my main machine, and 3 other computer on the team render... well, maybe I'm wrong on that... actually, my 2nd machine has Corona5 installed, is on net, but asy "No license loaded", and I don't know how to "give" it the license... if somebody can make my ideas clearer??? thank you!!!

The article at https://help.c4d.corona-renderer.com/support/solutions/articles/12000013779-how-to-use-team-render-distributed-rendering- covers that in the section "Activating Corona using the Licensing Server"
Tom Grimes | chaos-corona.com
Product Manager | contact us

2020-01-13, 16:29:04
Reply #10

houska

  • Former Corona Team Member
  • Active Users
  • **
  • Posts: 1512
  • Cestmir Houska
    • View Profile
I'm not sure if it's a valid question:  I thought that, with the monthly fee, I get Corona on my main machine, and 3 other computer on the team render... well, maybe I'm wrong on that... actually, my 2nd machine has Corona5 installed, is on net, but asy "No license loaded", and I don't know how to "give" it the license... if somebody can make my ideas clearer??? thank you!!!

Also, you can now activate manually using the Machine preferences in the TR client app. See the attached image!
« Last Edit: 2020-01-13, 16:42:19 by houska »

2020-02-03, 14:50:14
Reply #11

blewis

  • Users
  • *
  • Posts: 3
    • View Profile
My Team Render seems to be working fine, but anyone know why I render locally and a frame takes 6.5 minutes, but when I turn on Team render, the same computer takes 16.40 minutes.

2020-02-03, 14:55:35
Reply #12

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5434
    • View Profile
Taking longer with TR usually comes down to the network settings - things get held up on sending data between the computers, if data is sent too often and in chunks that are too large and the network becomes congested. First thing to try is setting the Team Render configuration in the Render Settings to Arbitrated and see if that helps. Next step would be to increase the client interval (send packets less often) and increase the Max packet size (when sending, send more information at once). I usually try things like every 20 or 30 seconds, with packet sizes of 128 or 256.
Tom Grimes | chaos-corona.com
Product Manager | contact us

2020-02-04, 18:51:35
Reply #13

prince_jr

  • Active Users
  • **
  • Posts: 83
    • View Profile
yes...teamrender needs time to copy assets. but in my case that's not the big part. the big part is creating the initial pass. it needs a lot of time! sometimes up to 30mins. (3333x5000px)
main machine...mac pro 12core 3.46ghz 64gb, renderslave mac pro 8 core 2.4ghz 32gb.
and couldn't figure out yet, what these chunks are doing. the bigger the resolution of the rendered image, the more chunks...i guess?
where are the setting options you are talking about (client interval etc.) in c4d?

2020-02-04, 18:55:11
Reply #14

TomG

  • Administrator
  • Active Users
  • *****
  • Posts: 5434
    • View Profile
In the Team Render tab in the Corona Render Settings, attached image.

By "Creating the initial pass", do you mean calculating the UHD Cache, or do you mean completing pass 1 of the render?
Tom Grimes | chaos-corona.com
Product Manager | contact us