[solved] DR machines start and fail loop

[Vuk]

In middle of a deadline for some reason DR is in a loop now I tried it from my workstation, tried it from a node as a backburner server job with dr for other slaves same thing. DR machines are green as soon as the render starts and then simultaneously start going red and green and red so they won't start. The scene is heavy around 1.8 gigs but I rendered the same scene without any adjustments last night from my workstation and all was working fine until this morning when I tried again and the problem started. I even cleaned the scene up got it down to 1.2 gigs and the problem persists.

I am sure its not a network problem since all was working fine less then 12 hours ago. There was no Win update in the past days so that is excluded as well... Anyone experiencing the same problems?

[Frood]

I had DR loops back in v1.6.1 but never again since then. Which version is this?

Something in the logs? (slaves Max.log / DrLog.txt)

BTW: This sounds urgent, you may contact support through helpdesk:

https://coronarenderer.freshdesk.com/support/tickets/new


Good Luck

[Vuk]

Version 3.0 newest possible version. For some reason it just started working... Looking good so far fingers crossed got about 18 5k renders to render till the end of the day :) its gonna be tight!

[Frood]

Oh - glad to read! Favourable occasion for another


Good Luck ;]

[Vuk]

Ok so again the same problem in DR. Machines are parsing scene then starting then again parsing scene and go red connection failed.

Worked before without any issues in the same scene before I upgraded to 3.0. I had RC 4 or 5 of version 3.0 before and never had this issue. The scene is not even taking a 4th of total ram of the machines which are more or less all running with 64gb of ram. All the pats are UNC and from a Server. I had this issue less then 2 weeks ago I was restarting nodes, scene trying it for 2 hours and then all of a sudden it started working. Now I am again on a deadline for the same images and again same problem... The only solution now seems to just start render by render on each machine...

[Juraj]

Just like Frood mentioned, I used to suffer from this all the time, but it was some time ago, pre-v2 version.

I wonder if it's some particular feature in scene that sets it off, because I would have it happen only on every second scene, it's wasn't constant thing. Since all my scenes are kinda super heavy, it was impossible to pin-point by myself (or I never had the patience).

[Vuk]

Actually scene was kinda heavy the first time it happened and I got it down from 2gb to 1.2gb there is no plugin nothing in particular just floorgenerator, multitexture all is corona did find some vray displacement trough Forensic and deleted it but it still does the same thing. Funny thing is that this actually started the day 3.0 was released and I upgraded from RC to full version of 3.0...

[Frood]

Maybe rebooting slaves may help?

https://forum.corona-renderer.com/index.php?topic=22767.0

Not that I'm a fan of such kind of "solutions" but you take what you get when it's getting urgent.


Good Luck

[Vuk]

Tried that as well a few times actually even rebooting the OS. No help, tried rendering from different computer same thing. Rendered another scene which is the same project but a different scene with half of the same assets and it works. Apparently its something inside the scene that its bothering the DR. I guess that maybe merging all into a new scene would solve the problem but I was in such a rush that I just went with single machine rendering for each render...

[Jpjapers]

Ive experienced this recently with v2 and im certain its something to do with scene size as its only ever with a heavy scene and usually works the first time and then fails every time after.

[Vuk]

I am pretty sure it hasn't got anything to do with scene size. Its a simple house/interior scene the file is less then a gb... I have done several aerial/masterplans lately and worked with much much bigger and complex files and never had this issue.
I tried merging the complete scene in a new max file and the problem persists. I am now pretty sure that this is related to a single object/model or asset that I merged but since there are so many and the deadline is tight I have completely given up on it.

[Vlad_the_rant]

This is probably caused by a malformed object somewhere in the scene or a misbehaving plugin.
An option to try would be to convert as many objects to editable meshes as possible. Applying a "reset xform" before conversion to editable mesh could also help, the aim being to have all objects be just meshes, with no modifier stacks and at the correct scale since Max can and does sometimes freak out when something is wrong with the modifier stack.
Why, you may ask? Because when a Max file is loaded, what actually happens is Max rebuilds everything. This means taking the base object and applying all the modifiers to it with their parameters during loading, so when an object is just a mesh with no modifier stack, it will just load the mesh and there's less chance some modifier or something can get screwed up during the loading of the scene.

[Juraj]

That sort of searching is basically impossible with production scene with millions of objects.

Annoying culprit of this behavior is that those scenes render just fine themselves on both workstations or individual slaves. They only fail in DR mode on slaves.

I don't remember which scene of mine did it but maybe VUK could upload his (though I am sure I uploaded mine as well back then).

[Jpjapers]

That sort of searching is basically impossible with production scene with millions of objects.

Annoying culprit of this behavior is that those scenes render just fine themselves on both workstations or individual slaves. They only fail in DR mode on slaves.

I don't remember which scene of mine did it but maybe VUK could upload his (though I am sure I uploaded mine as well back then).

Agreed, it works fine on my workstation too.
I dont want to have to collapse my stack just to render something on the farm reliably. Its far too destructive a workflow.

The solution to the problem shouldn't be to change the way we work.

[Vlad_the_rant]

The solution to the problem shouldn't be to change the way we work.

Agreed, but if the problem is sporadic and there's a tight deadline, collapsing everything may be the only option, just for that particular job.

[Jpjapers]

I'll concede to that

[Vuk]

As soon as I am out of the deadline I will check if that solves the problem, but to be honest in my working environment this is a rather light scene and I worked on much heavier scenes with loads of stacks on geometry and never had this issue...

[Frood]

This one seems to be a similar issue, if not the same:

https://forum.corona-renderer.com/index.php?topic=22816.0


Good Luck

[Vuk]

Long time no see and again me:) Hadn't had the time to play with the scene but I did a few projects after and everything was working fine was actually working on a new project and for several days all was working just ok even last night I was rendering with nodes without any issues till this morning when I wanted to re-render an image and all of a sudden I am stuck at parsing scene its not moving I tried just with my workstation tried with nodes not moving just parsing endlessly. I have upgraded to version 3.1 last week.
I don't get it was working last night I haven't even merged a single model or object to the scene. Nothing new was introduced just the same scene that was working yesterday will not render now at all.

When I try to stop it the green progress bar moves till the end and the process is stuck no reaction I can only end task the program no way to stop it the normal way by closing it. From when I upgraded to 3.0 its just issue after issue after issue... I am kinda disappointed...

[maru]

@Vuk - have you ever contacted us about your issues through our support portal? (https://coronarenderer.freshdesk.com/support/tickets/new)
DR issues are very hard to diagnose, especially in cases like this where it randomly works and doesn't.
Some clues you may investigate:
1) Missing plugins on the slave PCs?
2) Different plugin versions on the slave PCs?
3) Some issue with the network? (Are you able to send a very large file from one computer to another, without interruption?)
4) Windows Update messed something up? (one day it was working, another day it was not kind of theory)
5) Firewall/antivirus blocking something? (best way to check this is by disabling all firewall/antivirus software on you master and slave PCs - including Windows Firewall and Defender)

The best idea would be to contact us though the link above and send us the problematic scene with all assets included (see my signature for the uploader). We would then try to reproduce it, and meanwhile think of other things to check.

[Vuk]

@Maru

So again the same problem yesterday. I went trough your list this time since I haven't bothered to check the forum for a while and yesterday I saw your post. Thnx for all the points you gave. Turns our I am not completely sure but it is Win update related. Yesterday the computers started shutting down one by one and on some of the workstations the update didn't go trough so today I did a manual check update and wuala it works now.

Hope it stays like this now and the issue was win 10 update related. For some reason when I was on Corona 2.0 this issue never happened. Seems that 3.0 is much more sensible to win 10 updates. And god I hate win 10 more and more...

Thnx again!

[maru]

This might be the solution. It was randomly found by one of our users, and it fixed the problem for them. That's why it's so important to share solutions with the world. :)


Related to Windows 10 fall update and SMBv1 change https://support.microsoft.com/en-us/help/4034314/smbv1-is-not-installed-by-default-in-windows

1) Go to the Windows Services applet, find "Function Discovery Provider Host" and "Function Discovery Resource Publication" services, right click on them one by one, and set them to Automatic (Delayed Start), and then press Start.

2) Make sure network discovery is enabled for all of your computers: https://www.windowscentral.com/how-configure-network-discovery-windows-10-0