Chaos Corona Forum

Chaos Corona for Cinema 4D => [C4D] General Discussion => Topic started by: friedli3d on 2019-01-25, 07:59:20

Title: Team Render Job failed
Post by: friedli3d on 2019-01-25, 07:59:20

Hello,

I have purchased a corona license for c4d and im trying to make to work with team render but im having an issue.

The error says (Error) Render-Job failed: Rendering stopped because of an out-of-memory or unknown error

I have tried with a different file, with simple materials, and the team render worked fine, but with this file, which has a lot of objects, refs, etc, it gets always this error.

I send the message from console, with a render that worked fine and the one who doesn't.

Thank you for your help


    > 3designs-iMac-Pro1.local. Download-Speed 14.73 MiB\s (1x)
Peer-to-Peer Statistics End
Grabbed Render-Job 'pv' from machine '3designs-iMac-Pro1.local.'
Created Asset 'Untitled 1.c4d'
Downloaded Asset(s) in 0.009 seconds
Start Rendering for Machine 3designs-iMac-Pro1.local.
Rendering frame 0 of job 'pv'
2019/01/25 07:32:46  [Corona4D] Parsing scene
2019/01/25 07:32:46  [Corona4D] ====RENDER STARTED====
2019/01/25 07:32:46  [Corona4D] Core build timestamp: Jan 18 2019 18:35:55
2019/01/25 07:32:46  [Corona4D]

Sending HELLO to server 2019/01/25 07:32:46  [Corona4D] Connection established 2019/01/25 07:32:46  [Corona4D] REPORT: unchanged: 0, Prim: 400, Inst: 0, Mtl: 0, Env-tentative: 1 2019/01/25 07:32:46  [Corona4D] Core::renderFrame before onFrame 2019/01/25 07:32:46  [Corona4D] Parsing scene took 0.166 seconds 2019/01/25 07:32:46  [Corona4D] Calculating displacement 2019/01/25 07:32:46  [Corona4D] Preparing geometry took 0 seconds 2019/01/25 07:32:46  [Corona4D] Preparing geometry 2019/01/25 07:32:46  [Corona4D] Embree: memory usage: 21.2 MB. Total embree build time is 65ms, top level commit took: 59ms. There are 400 embree geometry groups, 0 instances, 400/0/0/0 tri/animTri/hair/custom primitives 2019/01/25 07:32:46  [Corona4D] Core::renderFrame after onFrame 2019/01/25 07:32:46  [Corona4D] CoronaCore::renderFrame before unique materials enum 2019/01/25 07:32:46  [Corona4D] CoronaCore::renderFrame after unique materials enum 2019/01/25 07:32:46  [Corona4D] CoronaCore::renderFrame after unique materials update 2019/01/25 07:32:46  [Corona4D] Preparing geometry took 0.155 seconds 2019/01/25 07:32:46  [Corona4D] Preparing lights 2019/01/25 07:32:46  [Corona4D] TreeAdapt built: nodes 0, time 0 ms, memory 0.088 kB 2019/01/25 07:32:46  [Corona4D] CutCache initialized: records 299499, time 4 ms, memory 11.9802 MB 2019/01/25 07:32:46  [Corona4D] CoronaCore:RenderFrame: after directLight 2019/01/25 07:32:46  [Corona4D] Preparing lights took 0.005 seconds 2019/01/25 07:32:46  [Corona4D] Computing sec. GI 2019/01/25 07:32:46  [Corona4D] CoronaCore::renderFrame: after GI precompute 2019/01/25 07:32:46  [Corona4D] Computing primary GI took 0.052 seconds 2019/01/25 07:32:46  [Corona4D] CoronaCore::renderFrame: before render 2019/01/25 07:32:46  [Corona4D] Rendering initial pass 2019/01/25 07:32:46  [Corona4D] Rendering pass 3 2019/01/25 07:32:47  [Corona4D] Rendering pass 6 2019/01/25 07:32:47  [Corona4D] Sending chunk 0/1 to the server 2019/01/25 07:32:47  [Corona4D] Rendering initial pass 2019/01/25 07:32:47  [Corona4D] Rendering pass 2 2019/01/25 07:32:48  [Corona4D] Sending chunk 0/1 to the server 2019/01/25 07:32:48  [Corona4D] Rendering initial pass 2019/01/25 07:32:49  [Corona4D] Rendering pass 2 2019/01/25 07:32:49  [Corona4D] Rendering pass 3 2019/01/25 07:32:50  [Corona4D] Rendering pass 4 2019/01/25 07:32:50  [Corona4D] Rendering pass 2 2019/01/25 07:32:50  [Corona4D] Sending chunk 0/1 to the server 2019/01/25 07:32:51  [Corona4D] Rendering pass 4 2019/01/25 07:32:51  [Corona4D] Rendering pass 9 2019/01/25 07:32:51  [Corona4D] Rendering pass 11 2019/01/25 07:32:52  [Corona4D] Rendering pass 14 2019/01/25 07:32:52  [Corona4D] Rendering pass 15 2019/01/25 07:32:52  [Corona4D] Rendering pass 16 2019/01/25 07:32:53  [Corona4D] Sending chunk 0/1 to the server 2019/01/25 07:32:53  [Corona4D] Rendering pass 2 2019/01/25 07:32:53  [Corona4D] Rendering pass 3 2019/01/25 07:32:53  [Corona4D] CoronaCore::renderFrame: after render 2019/01/25 07:32:53  [Corona4D] Terminating DR slaves 2019/01/25 07:32:53  [Corona4D]  - terminating slave handlers 2019/01/25 07:32:53  [Corona4D]  - waiting for broadcast thread to finish 2019/01/25 07:32:53  [Corona4D]  - clearing slave handlers 2019/01/25 07:32:53  [Corona4D] Terminating DR slaves ended 2019/01/25 07:32:53  [Corona4D] Rendering took 7.303 seconds 2019/01/25 07:32:53  [Corona4D] Cleaning up 2019/01/25 07:32:53  [Corona4D] Terminating DR slaves 2019/01/25 07:32:53  [Corona4D]  - terminating slave handlers 2019/01/25 07:32:53  [Corona4D]  - waiting for broadcast thread to finish 2019/01/25 07:32:53  [Corona4D]  - clearing slave handlers 2019/01/25 07:32:53  [Corona4D] Terminating DR slaves ended 2019/01/25 07:32:53  [Corona4D] CutCache memory: 12.1504 MB 2019/01/25 07:32:53  [Corona4D] Unique Primitives: 297432 2019/01/25 07:32:53  [Corona4D] Primitives with instancing: 297432 2019/01/25 07:32:53  [Corona4D] Area lights: 2 2019/01/25 07:32:53  [Corona4D] Geometry groups: 400 2019/01/25 07:32:53  [Corona4D] Materials: 1 2019/01/25 07:32:53  [Corona4D] Instances: 400 2019/01/25 07:32:53  [Corona4D] Portals: 0 2019/01/25 07:32:53  [Corona4D] Area lights: 2 2019/01/25 07:32:53  [Corona4D] Avg samples per pixel: 1.23473 2019/01/25 07:32:53  [Corona4D] Avg rays per sample: 20.4952 2019/01/25 07:32:53  [Corona4D] Rays/s: 3.17091e+06 2019/01/25 07:32:53  [Corona4D] Samples/s: 155816 2019/01/25 07:32:53  [Corona4D] UHDCache records: 805 2019/01/25 07:32:53  [Corona4D] UHDCache records added during viz: 0 2019/01/25 07:32:53  [Corona4D] UHDCache rejected %: 71.8781 2019/01/25 07:32:53  [Corona4D] Saving + Cleaning up took 0.002 seconds 2019/01/25 07:32:53  [Corona4D] CoronaCore::exiting renderFrame 2019/01/25 07:32:53  [Corona4D] Rendered 3/0 passes 2019/01/25 07:32:54  [Corona4D] Sending chunk 0/1 to the server 2019/01/25 07:32:54  [Corona4D] Finished chunks: 1/1 Peer-to-Peer Statistics:     > 3designs-iMac-Pro1.local. Download-Speed 14.12 MiB\s (1x) Peer-to-Peer Statistics End Grabbed Render-Job 'pv' from machine '3designs-iMac-Pro1.local.' Created Asset 'hanni_areal_3dmodel.c4d' Created Asset 'Corona C4D.c4d' Created Asset 'EVOQUE.c4d' Created Asset 'vm_v6_016_corona.c4d' Created Asset 'VP_Free_MazdaMX5_corona.c4d' Downloaded Asset(s) in 0.901 seconds Start Rendering for Machine 3designs-iMac-Pro1.local. (Error) Render-Job failed: Rendering stopped because of an out-of-memory or unknown error Peer-to-Peer Statistics:     > 3designs-iMac-Pro1.local. Download-Speed 42.43 MiB\s (8x) Peer-to-Peer Statistics End

Title: Re: Team Render Job failed
Post by: TomG on 2019-01-25, 14:30:52

Be sure to either start the Corona License Server first before the TR Clients, or best yet, update to the latest daily which resolves the need to do that (see https://forum.corona-renderer.com/index.php?topic=23346). Let us know if either of those resolve the issue. It could be of interest to have the scene that causes problems too, since it doesn't happen in every scene - you can use the private uploader. See https://help.c4d.corona-renderer.com/support/solutions/articles/12000033461-how-to-report-issues- for how to report issues and send in things via the uploader (like TR logs, scene, etc.)

Thanks!

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-01-28, 11:52:26

Thank you very much for your help!

yes the (Error) Render-Job failed: Rendering stopped because of an out-of-memory or unknown error is not showing anymore, and it seems to work now.

However some xref objects are with some transparency and the team render is actually more slow than render in only one machine , but I think that may be a problem relative to Maxon.

Once again, thank you for your help and next time I will follow better how to report the issue :)

Title: Re: Team Render Job failed
Post by: TomG on 2019-01-28, 13:59:52

Hi!

Some quick thoughts - I've only found TR slower than on one machine when the packet size is too small (it may happen with larger resolutions), and raising that by setting the TR settings in the Corona render settings to manual and using a larger packet size corrects that. What is the resolution of the image?

Other than that, it may be the xrefs are on a network drive, not sure how that might impact things if so? It could be a C4D issue, but might be interesting to know more just in case :)

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-01-31, 11:22:42

You are totally right!

The ouptut size is actually not so big ( 2560x1600 ) but when I changed the size of packet, made all the difference in time difference.

So, with client interval 5s and maximum size packet 2, the render without TR was 00:15:41 and with TR was 00:18:21, so with TR took more time.

When I changed the size of packet to 128mb the render with TR was 00:08:19, so almost half the time, that was awesome!

However I still have a problem, if I put in size packet lets say, 256mb or 512mb, the render won't finish totally.

I attach a file with the issue

This may seem stupid but is there any relation between size packet and output size?

thanks again for your help

Title: Re: Team Render Job failed
Post by: TomG on 2019-01-31, 14:12:22

Hi!

Can you give more information about the "won't finish totally" - was it rendering to noise, passes, or time? How many passes did it say it had done, compared to how many it does on the local machine only? Did it finish receiving all data from the clients (ie was it really finished)? Does it just stop normally, or with any sort of error (either directly in C4D, or shown in the logs in the clients)? What does the image look like from just the local machine?

The scene might be useful in this case so that we can test here, since that sounds a bit unusual, can always use the private uploader to send it along, info for that is on the https://help.c4d.corona-renderer.com/support/solutions/articles/12000033461-how-to-report-issues- page (along with other useful info :) )

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-01-31, 15:18:42

Hi!

Sure, so I have max 30 passes with denoising and im also using LightMix. The render appears as finished, so it completes 30/30 passes without any warning or stop, but the final looks like the image I sent.

I will send another two images, one with and without TR with size packet at 256mb. Both with denoising and 30 passes completed

I will also upload the file throw private uploader.

Thank you once again

Title: Re: Team Render Job failed
Post by: TomG on 2019-01-31, 15:40:24

Definitely looks like it hasn't received all the data from the Clients, which is why the top half looks perfect (all data received), and the bottom is just an early pass. With LightMix layers being included, I wonder if it wasn't still receiving data and hadn't finished that step (but the render would be done, so the VFB would look like it was done)?

Title: Re: Team Render Job failed
Post by: houska on 2019-01-31, 16:58:42

Hi, did you by any chance hit the stop button twice?

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-01, 09:39:44

Hi all,

I assume the render was finished because it appears the green light in picture viewer when the render is finish.

I tested without the LightMix and the render is ok, so I assume is something relative to LightMix.

@houska , no I haven't hit the stop button twice, not even once, I let the render finish :)

I already sent the file throw private uploader

Thank you all

Title: Re: Team Render Job failed
Post by: Beanzvision on 2019-02-01, 10:40:02

Quote from: friedli3d on 2019-02-01, 09:39:44

Hi all,

I assume the render was finished because it appears the green light in picture viewer when the render is finish.

I tested without the LightMix and the render is ok, so I assume is something relative to LightMix.

@houska , no I haven't hit the stop button twice, not even once, I let the render finish :)

I already sent the file throw private uploader

Thank you all

Hi, do you recall what the file is called that you uploaded, was it to dropbox or the regular uploader?

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-01, 11:37:09

hi,
the file is hanni areal 3dmodel.zip

I may send it twice, because took a while to send and I was not sure if it send it. the file is not to heavy, 173.5mb

thank you

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-01, 13:54:45

Not sure if the green light simply means "All rendering calculations are finished" - it may not mean "and all data has been sent over the network from the slaves and combined with the original". There could be a lot of data to move with LightMix elements. What happens if you let it sit there for 5 minutes after the green light?

PS also, for the devs, do you know when the light goes green here, on calculations being finished only, or on "all data received and combined and displayed"? :)

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-01, 15:40:44

Well, I don't know why but now it works correctly, the render with and without Team Render.
I just deleted the xref that a I had,and then insert them again, because thoose xrefs were showing some transparency, and now it works fine.

But when the final render with TR was showing the problem, if I waited 5/10m the problem maintains.

Not sure if I understood the other question. I assume that when the light circle in picture viewer and also on TR Client is green all data received and combined and displayed are completed.

Do you already have opportunity to test it with the file I sent?

Thank you!

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-01, 16:01:46

I can't test on a Mac, I am Windows only :) So that would be for the developers to have a look into.

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-01, 16:03:01

Oh unless by "green light" you are referring to the status of the machine in the TR Machines list?

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-01, 16:14:08

As a PS, I don't see the scene in either dropbox or private uploader.

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-04, 08:12:21

ah, ok!  Thanks anyway :)

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-04, 13:44:13

Yes, green light I was referring also to the TR machines list

the file has this name :

1549283860_hanni-areal-3dmodel.zip


Thank you!

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-04, 15:37:29

File is there now, ty.

As a note, I have the following warning about missing plugins, MultiCloner, MultiFlora and Forester Tree, so any testing I am able to do will have to keep that in mind. Not sure about what others in the support team have installed.

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-04, 15:47:20

Yeah, I can't use TR as it just quits due to the missing plugins, sorry.

Title: Re: Team Render Job failed
Post by: Beanzvision on 2019-02-04, 22:28:43

I have Forester but won't be able to test it until the morning I'm afraid.

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-05, 08:47:44

No problem at all!

thank you very much for your help guys

Title: Re: Team Render Job failed
Post by: Beanzvision on 2019-02-05, 11:34:07

Hi, So I have rendered your scene file and also had a memory warning but from my slaves. It did finish rendering though. I wasn't surprised though because there are a large number of detailed trees on the horizon that when rendered you can't really tell if they are high quality or 2D. Personally, I would only render what is needed, if it's not in the camera view then why should it be there eating up the memory? Obviously, for closer shots, this would be different. Render time was 26mins on 4 nodes.

(https://forum.corona-renderer.com/index.php?action=dlattach;topic=23356.0;attach=97985;image)

Title: Re: Team Render Job failed
Post by: friedli3d on 2019-02-05, 13:38:59

Thank you beanzvision

Yes, you are right, although i never got that warning i should do that and reduce the detailed trees in background.

Just one more question. in the test you have made, there is a car that is almost transparent, do you know that is happening?

I tried to ask to maxon, but i still haven't any answer.

That only happens to me when i use Team Render and my scene file is saved on the server. If i save the project with assets to my desktop and render from there with TR, works fine.

Thanks again

Title: Re: Team Render Job failed
Post by: Beanzvision on 2019-02-05, 13:55:27

Hi, I just noticed that now. I'm not entirely sure why this would be happening. It would be interesting to see if it happens with the standard renderer also. If it does then I would say it's a Maxon issue, If it's just with Corona then we will need to investigate further.

Title: Re: Team Render Job failed
Post by: TomG on 2019-02-05, 14:19:45

When something appears as transparent in a network rendering situation, it usually means the nodes did not receive it, so they rendered the image without it, which was combined with the image from the master machine that did have it. This might be related to being an XRef or other clone, or object that is on a network drive or on a local drive that the client machines didn't have access to, etc.