Team Render has been completely unusable

[shane_w]

TeamRender has been extremely unreliable using Corona over the past few releases.  I don't know if something has changed in the builds or on my system but I have problems running even the simplest jobs. Projects run perfectly with the native render and Vray.  Projects seem to fail near the end and usually can be restarted multiple times to get it to finish.  Previous builds would crash the clients but I don't see that anymore with 08-29 build they just drop the job, all clients go idle reporting the same error.

(Error) Render-Job failed: Rendering stopped because of an out-of-memory or unknown error
2017/08/29 13:58:22  Cancelled because all attending machines failed
2017/08/29 13:58:22  Render Failed

I have re-installed the server and clients, re-installed Corona, removed all plugins and Vray.  I even rolled everything back to a previous build that used to work flawlessly.  Nothing seems to help fix the issue, It worked very well until about a month ago and I have not been able to sort out what has changed.

system specs:
1 machine running Server app
8 render machines running client app
TeamRender 18.057
Corona 08-29-2017
OS X 10.12.6


Any ideas or suggestion would be appreciated.

Thanks,

-Shane

[houska]

TeamRender has been extremely unreliable using Corona over the past few releases.  I don't know if something has changed in the builds or on my system but I have problems running even the simplest jobs. Projects run perfectly with the native render and Vray.  Projects seem to fail near the end and usually can be restarted multiple times to get it to finish.  Previous builds would crash the clients but I don't see that anymore with 08-29 build they just drop the job, all clients go idle reporting the same error.

(Error) Render-Job failed: Rendering stopped because of an out-of-memory or unknown error
2017/08/29 13:58:22  Cancelled because all attending machines failed
2017/08/29 13:58:22  Render Failed

I have re-installed the server and clients, re-installed Corona, removed all plugins and Vray.  I even rolled everything back to a previous build that used to work flawlessly.  Nothing seems to help fix the issue, It worked very well until about a month ago and I have not been able to sort out what has changed.

system specs:
1 machine running Server app
8 render machines running client app
TeamRender 18.057
Corona 08-29-2017
OS X 10.12.6


Any ideas or suggestion would be appreciated.

Thanks,

-Shane

Hi Shane!

As you're probably aware, the latest build came with a lot of fixes and changes to the Team Render, so it is only expected to have a bug or two. But don't worry, we'll look at these bugs and do what we can. Just for clarification:

Are all your machines Macs? Or do you have Windows clients and one Mac server?
Does the issue happen with all the scenes?
Also, do I read it correctly that now even the older builds fail for you?

Thanks for your patience!

Cestmir

[Nikola]

Hi Shane,

Could you please try to render simple scene directly with Cinema 4D (not the CINEMA 4D TeamRender Server) with only one client and then send me the output of Cinema 4D Console (Script -> Console) and the output of the CINEMA 4D TeamRender Client Console (Console Tab). Thank you

Nikola

[shane_w]

Hi Shane!

As you're probably aware, the latest build came with a lot of fixes and changes to the Team Render, so it is only expected to have a bug or two. But don't worry, we'll look at these bugs and do what we can. Just for clarification:

Are all your machines Macs? Or do you have Windows clients and one Mac server?
Does the issue happen with all the scenes?
Also, do I read it correctly that now even the older builds fail for you?

Thanks for your patience!

Cestmir

All machines are Mac

Seems to happen with all scenes.  I even made some very simple scenes for testing.

Yes it started happening with older builds.  That is the most troubling part of it.  It started right around the time of the last update from Maxon.  I even contacted Maxon and rolled my Cinema install back to the previous version.  There was even a OS X update at the time so I have had a very hard time trying to figure out what is causing it.

I didn't report it sooner as I was hoping the work on Team Render in the current build might make an improvement.

Thanks,

-Shane

[shane_w]

Could you please try to render simple scene directly with Cinema 4D (not the CINEMA 4D TeamRender Server) with only one client and then send me the output of Cinema 4D Console (Script -> Console) and the output of the CINEMA 4D TeamRender Client Console (Console Tab). Thank you

Ok, I think I understand what you are asking for.

This is what I did..... created a simple scene > enabled one machine in Team Render Machines > Team Render to Picture Viewer > results attached

Thanks,

-Shane

[Nikola]

Could you please try to render simple scene directly with Cinema 4D (not the CINEMA 4D TeamRender Server) with only one client and then send me the output of Cinema 4D Console (Script -> Console) and the output of the CINEMA 4D TeamRender Client Console (Console Tab). Thank you

Ok, I think I understand what you are asking for.

This is what I did..... created a simple scene > enabled one machine in Team Render Machines > Team Render to Picture Viewer > results attached

Thanks,

-Shane

Thank you for the results. According to them, it looks that render finished successfully. Did Cinema 4D report any error?

[shane_w]

No error.  But all of my problems are when submitting a job to my render farm and using Team Render Server & Clients.  I never use "Team Render to picture viewer" on my local machine.

Thanks,

-Shane

[shane_w]

Exact same scene fails when rendered using Server / Clients.

[Nikola]

According to your screenshot, it looks that Corona is not loaded in the Team Render Server. You can check it in your Team Render Server Web UI on Network page after clicking on Plugins section. If there is no Corona plugin listed then start Team Render Server with following command in the Terminal and send me the output please:

Code: [Select]

/Applications/MAXON/CINEMA\ 4D\ R18/CINEMA\ 4D\ TeamRender\ Server.app/Contents/MacOS/CINEMA\ 4D\ TeamRender\ Server
Nikola

[shane_w]

Ok, I did not have the Corona plugin installed in the server app.  Once I put it in the plugin folder I am now able to render a single frame through Team Render server + clients.  But instead of one machine taking the single frame and rendering it all machines render it together.  It took my 8 servers roughly the same amount of time to render the image as it did rendering it on my workstation???

Is there no way to send a single frame and have only one server render it?

I am back to my original problem, if I send a 20 frame animation only one server will pickup some frames.  And once it completes some frames the job will fail this happens over and over.  Start it again, render a few frames fail again.  I have attached a log showing me trying to render a 20 frame sequence.

Thanks,

-Shane

[Nikola]

Ok, I did not have the Corona plugin installed in the server app.  Once I put it in the plugin folder I am now able to render a single frame through Team Render server + clients.  But instead of one machine taking the single frame and rendering it all machines render it together.  It took my 8 servers roughly the same amount of time to render the image as it did rendering it on my workstation???

Is there no way to send a single frame and have only one server render it?

I am back to my original problem, if I send a 20 frame animation only one server will pickup some frames.  And once it completes some frames the job will fail this happens over and over.  Start it again, render a few frames fail again.  I have attached a log showing me trying to render a 20 frame sequence.

Thanks,

-Shane

Hi Shane,

Today I successfully reproduced your problem. Team Render in Cinema 4D has two modes. The first mode is Team Rendering of animations when each client gets some range of frames which are rendered by normal (local) renderer one after another and finished frames are sent back to the server (all communication is handled by Cinema for us). The second mode is called "Single frame Team Rendering"  which is started by Cinema when only one frame is being rendered (no animation). In this mode, each client renders the same image, but with a different seed, i.e. different pass and all these passes are sent back to the server and combined to produce final render. In this mode, we must handle all communication (but not only communication but also saving, nearly everything) by ourselves and this is what I was mainly fixing in the last daily build. Despite the Cinema 4D is handling the animation Team Rendering for us, there is probably some bug in local renderer so I will try to find the cause of it today.

Nikola

[shane_w]

Thanks so much for your help Nikola, I am not sure what bug you reproduced as I am having multiple problems.

To be clear my main and most severe problem is when rendering animations using Team Render server and clients.  My local machine is not involved at all.  I am saving project and assets and placing them on my server and starting the rendering using the Team Render server web app.  When I do this I can not get a job to finish.  It doesn't seem to assign the frames correctly to the clients and once they finish rendering a frame or two the just go offline.

Thanks,

-Shane

[Nikola]

Thanks so much for your help Nikola, I am not sure what bug you reproduced as I am having multiple problems.

To be clear my main and most severe problem is when rendering animations using Team Render server and clients.  My local machine is not involved at all.  I am saving project and assets and placing them on my server and starting the rendering using the Team Render server web app.  When I do this I can not get a job to finish.  It doesn't seem to assign the frames correctly to the clients and once they finish rendering a frame or two the just go offline.

Thanks,

-Shane

I successfully reproduced the problem when rendering jobs are not able to finish. But why are you saving the whole scene and using the Team Render Server web application? For me, it looks easier to turn your Team Render Server to ordinary Team Render Client and simply use the "Team Render to Picture Viewer" (when the bug with unfinished jobs will be fixed). You can also use your main machine as the client by enabling "Edit -> Preferences -> Renderer -> Team Render -> Share Machine over Network".

Nikola

[shane_w]

But why are you saving the whole scene and using the Team Render Server web application? For me, it looks easier to turn your Team Render Server to ordinary Team Render Client and simply use the "Team Render to Picture Viewer" (when the bug with unfinished jobs will be fixed). You can also use your main machine as the client by enabling "Edit -> Preferences -> Renderer -> Team Render -> Share Machine over Network".

Great I'm glad you were able to reproduce the problem and I hope it is an easy fix for you. 

For our studio the web app is much better.  We have multiple people sending multiple jobs.  When a project is ready to render I have a little script that collets all assets saves them to the admin folder and I just need to press start in the web app and keep working.  My machine is not affected and the farm just renders our jobs.  This works great to queue up both still frames and animations and all users can view progress, quite jobs, change priority, delete jobs, get there final renderings etc.  Much more seamless and effective.   

I might use it the way you suggested if I was the only person rendering or if I have an urgent rendering that needs the combined power of the whole render farm.

Thanks,

-Shane

[Nikola]

Hi Shane,

the new daily build is out (https://forum.corona-renderer.com/index.php?topic=15602.msg108734#msg108734). Can you please test it whether it fixes your problems?

Nikola