Author Topic: Render Nodes Stopping  (Read 1882 times)

2019-11-20, 17:47:36

Br0nto

  • Active Users
  • **
  • Posts: 47
    • View Profile
Hi Corona Team,

We're running into a very frustrating issue that has popped up repeatedly over the past few months. We're running Corona Renderer for Cinema 4D, version 4 hotfix 3.

The issue appears in two parts:

1) We start a team render with 2 or 3 nodes running. As soon as we start an additional node, the others drop off and suddenly cease rendering. No error, no warning, just a total stop.

2) We then restart the other nodes, everything seems to be rendering, and we leave for the night. At a random point a few hours later, nodes begin stopping on their own. Same thing, no error, no warnings, only a complete halt to the process. Note that this particular render ended up doing a lot more passes than we intended, but that should have no bearing on the issue.

Here's what the log looks like in every case, regardless of scene, which computer begins the render, or how many nodes we have running:

Quote
2019/11/20 09:19:36  [Corona4D] [tr client] Rendering pass 533
2019/11/20 09:19:51  [Corona4D] [tr client] Rendering pass 534
2019/11/20 09:20:07  [Corona4D] [tr client] Rendering pass 535
2019/11/20 09:20:22  [Corona4D] [tr client] Rendering pass 536
2019/11/20 09:20:27  Service ALEXS-MAC-PRO went offline
2019/11/20 09:20:27  [Corona4D] [tr client] CoronaCore::renderFrame: after render
2019/11/20 09:20:27  [Corona4D] [tr client] Terminating DR slaves
2019/11/20 09:20:27  [Corona4D] [tr client]  - terminating slave handlers
2019/11/20 09:20:27  [Corona4D] [tr client]  - waiting for broadcast thread to finish
2019/11/20 09:20:27  [Corona4D] [tr client]  - clearing slave handlers
2019/11/20 09:20:27  [Corona4D] [tr client] Terminating DR slaves ended
2019/11/20 09:20:27  [Corona4D] [tr client] Rendering took 7878.98 seconds
2019/11/20 09:20:27  [Corona4D] [tr client] Cleaning up
2019/11/20 09:20:27  [Corona4D] [tr client] Terminating DR slaves
2019/11/20 09:20:27  [Corona4D] [tr client]  - terminating slave handlers
2019/11/20 09:20:27  [Corona4D] [tr client]  - waiting for broadcast thread to finish
2019/11/20 09:20:27  [Corona4D] [tr client]  - clearing slave handlers
2019/11/20 09:20:27  [Corona4D] [tr client] Terminating DR slaves ended
2019/11/20 09:20:27  [Corona4D] [tr client] CutCache memory: 123.907 MB
2019/11/20 09:20:27  [Corona4D] [tr client] Unique Primitives: 3910974
2019/11/20 09:20:27  [Corona4D] [tr client] Primitives with instancing: 3910974
2019/11/20 09:20:27  [Corona4D] [tr client] Area lights: 441816
2019/11/20 09:20:27  [Corona4D] [tr client] Geometry groups: 332
2019/11/20 09:20:27  [Corona4D] [tr client] Instances: 332
2019/11/20 09:20:27  [Corona4D] [tr client] Portals: 0
2019/11/20 09:20:27  [Corona4D] [tr client] Area lights: 441816
2019/11/20 09:20:27  [Corona4D] [tr client] Avg samples per pixel: 718.02
2019/11/20 09:20:27  [Corona4D] [tr client] Avg rays per sample: 38.8574
2019/11/20 09:20:27  [Corona4D] [tr client] Rays/s: 7.34272e+06
2019/11/20 09:20:27  [Corona4D] [tr client] Samples/s: 188969
2019/11/20 09:20:27  [Corona4D] [tr client] UHDCache records: 23
2019/11/20 09:20:27  [Corona4D] [tr client] UHDCache records added during viz: 0
2019/11/20 09:20:27  [Corona4D] [tr client] UHDCache rejected %: 98.4674
2019/11/20 09:20:27  [Corona4D] [tr client] Saving + Cleaning up took 0.019 seconds
2019/11/20 09:20:27  [Corona4D] [tr client] CoronaCore::exiting renderFrame
2019/11/20 09:20:27  [Corona4D] [tr client] Rendered 535/0 passes
2019/11/20 09:20:27  Service ALEXS-MAC-PRO went offline
2019/11/20 09:20:33  Peer-to-Peer Statistics:
    > Noahís iMac Download-Speed 39.28 MiB\s (4x)
    > ALEXS-MAC-PRO Download-Speed 44.73 MiB\s (5x)
    > Jeffís Mac Pro Download-Speed 47.63 MiB\s (5x)
    > desktop-t6m005k Download-Speed 73.57 MiB\s (1x)
Peer-to-Peer Statistics End

Some things we've noticed:

-This time, it was the two windows machines that dropped off, apparently right after they lost sight of the pilot machine (ALEXS-MAC-PRO). However, this has happened with macs before. Other nodes on the network continued rendering through the night, including ALEXS-MAC-PRO, which launched the render. We never had this issue before, and I have no idea what would have changed on our network that random machines can't see other machines.

-The additional cause-and-effect of nodes dropping when others are started, with the same log type, imply it isn't just a network issue.

Is there any way this could be a licensing issue? Is there some setting we should look at? This is really cutting into our production time in a dangerous way. Thanks!