Chaos Corona Forum

Chaos Corona for Cinema 4D => [C4D] Feature Requests => Topic started by: lollolo on 2018-10-26, 20:06:27

Title: RAM issue when using Team Render
Post by: lollolo on 2018-10-26, 20:06:27
Hello guys!

I work on MacOS 10.13.6 and C4D R20 and Corona DailyBuild Oct 15 2018. Another computer with windows 7 is for the rendering with Team Render.
-Started Team Render only on the pc so my Mac is free for other things.
-In the beginning C4D/Corona on my Mac used 6.7GB RAM.
-After I returned home and checked the finished render 36GB were in use and I got a warning
===== Warning(10004) =====
System is running low on RAM. You may experience excessive parsing times and/or crashes. You will probably have to simplify the scene.

22.1 GiB of physical RAM used by Cinema 4D
29.1 GiB of virtualized* RAM used by Cinema 4D
33 GiB of virtualized* RAM used by all running applications
32 GiB of physical RAM installed on system
33 GiB of virtualized* RAM provided by OS

* virtualized RAM means physical RAM + swap file on HDD/SSD

Corona RAM usage breakdown:
  229.4 MiB beauty/multi-pass

Team Render was deactivated on the Mac, the only task was to show the image with the picture-viewer. I had the same issue now several times.


Some other things I have seen:
-Crash while exporting corona proxy when the object has more than one uv tag

-Corona proxy is not working with Team Render.

-Mograph multi-instance is not working.

-Layer-Node get lost after restart of the application.

Thanks for your work!
Title: Re: RAM issue when using Team Render
Post by: houska on 2018-10-29, 09:43:51
Hello loliolg and welcome to the forums!

Thank you for your detailed report. So if I understand it correctly, your Mac computer had C4D open with your scene and you initiated the render from there, while your Windows computer was serving as a team render node? And you kept C4D open on your Mac and after several hours, the C4D application ate all your available RAM because of Corona? If that's the case, we'll need to investigate this issue of course.

Right now, I can only guess that there's either some sort of memory leakage somewhere or that the image viewer eats up all the ram while keeping the rendered images in memory (are you rendering an animation?).


About the Corona proxy issues: there's several things with it that we'll want to fix until the final release, so some issues are to be expected. Have you looked at this forum thread?
https://forum.corona-renderer.com/index.php?topic=14450
Maybe it could help you solve your problem with proxy in Team Render?

I'm not aware of any crashes while exporting a Corona proxy. It would be great if you could make a separate bug report here on the forums for that (thank you!)

The other bugs that you mention are already reported and known. In short - multiinstances are something new that R20 added and we'll need to write some code to support them, but it's planned. The node issues are also known and we're currently working on them.
Title: Re: RAM issue when using Team Render
Post by: houska on 2018-10-29, 10:08:06
One more remark regarding the node issues. Nik just told me that if you create a node material with layer shader in RC4 and save it, everything should be fine after loading. But if you save your scene in an older version and open the scene in RC4, there could be some issues. This is even mentioned in the RC4 changelog.

If the issue is happening completely in RC4 though, we'd love to have a separate bug report again. Thank you!
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2018-10-29, 12:54:35
Yes, you are right. That's the problem I have. It is increasing with the time, I can see it when I render high resolution images or an animation. Single frames which render only a few minutes are not problematic.
The example was a scene with many tree clones inside, rendered as a 4k image.

Also the cpu on my Mac is more active than when I use the native c4d render engine. I had some strange cases but I could't reproduce them yet.
I will make a post here when I know more.

Yes, I already know that thread.
Title: Re: RAM issue when using Team Render
Post by: HFPatzi on 2018-12-07, 09:25:08
Hi!

I experience almost the same issues with RAM usage in Teamrender. Except my RAM usage while rendering a high resolution still image with teamrender is almost instant at max. While rendering the same scene on my workstation only just uses around 3 to 4GB of RAM (see attached screenshots). Also nearly all my teamrender-clients crash while teamrendering this project. Unfortunatly without creating a bug report. The Teamrender client just quits.

I'm using the latest Corona Daily Build:

Full-speed non-debug version
Build timestamp: Oct 31 2018 17:15:05
Version: B2 (core 3)
Cinema version: CINEMA 4D Studio R18.057 S
Plugin status: initialized successfully


I have set a small render region in the cinema 4D render settings and set corona to render viewport selection only, singe i only needed this part of the image.

I'm sorry for not uploading the scene file because of nda. If you need any further informations, i'm happy to answer your questions.

Thanks for your Support!

Greetings from Germany,
Moritz
Title: Re: RAM issue when using Team Render
Post by: jojorender on 2018-12-16, 22:40:16
MacOs 10.13.6 / Build timestamp: Oct 31 2018 17:24:08 / Version: B2 (core 3) / CINEMA 4D Studio R18.057

Hi,
I have the same ram issue as OP.
This happens every time when team rendering. The ram usage creeps up over time until it reaches, in this case 149GB!!, and C4D becomes unresponsive. (See screenshots)
I have about 650GB available space on my boot drive, but the swap file size must have a “hard” limit.
Not sure what role Apple’s mysterious “kernel_task” plays in all this.

No multi-passes used (I always render multi-passes separately)
Just 1 render client used, on 1 Gb LAN
2 identical machines with 64GB ram
No other major process running.
Around the same time the “low mem” warning pops up, the client always disconnects “greenlights” in machine list.  Uncheck/check brings the client back online and renders for a bit, till it disconnects again…

Could this be some kind of memory leak in Team Render? I was able to save a CXR before C4D crashed, rebooted, “Resume from file” and continued rendering the image.
The render finished at the set 3% noise level  while only using around 27GB of ram the whole time.
“Resume from file” renders only on the main machine, a “Team Render from file” option would be great, once Team Render becomes really “useable”.

For me, the growing ram issue happens with every scene I teamrender, but depending on image size the render might just finish before c4d crashes. 3000x2000 is on the safe side, but 4500x3000 is pushing it.

I’m currently in a bit of a deadline crunch, but I can prepare a scene w/ assets once I have some time, if that helps. I assume I need to strip out all plugins like Laubwerk etc before sending.

Maybe, if the C-Team creates a sticky post with a sample scene that users can download to test TR, with instructions of what you need to see, might even give you a better insight into what’s happening.

Nevertheless, it’s time to say Thank You! for all the wonderful work you guys did so far with corona.
Hope that these team render issues can be ironed out before final release.

All my best.
Frank
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2018-12-18, 10:18:22
I rendered a new scene yesterday. I rendered again only on my render client and not on my main computer.
My render client used 28GB RAM while my computer (which didn't render) used around 60GB RAM (42GB C4D + 18GB kernel_task).

I hope you have any news soon!
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-06, 22:21:38
Do you have any news here? Can it be fixed in the near future?

It is quite hard to render high-res images with this bug. I had just another crash today morning. In this case 125GB RAM for C4D. Just too much for my iMac and C4D became unresponsive.
Title: Re: RAM issue when using Team Render
Post by: houska on 2019-01-07, 10:30:24
Hi loliolg

We fixed some issues with the proxy, which will appear in the next release. We also tweaked a few parameters of the Team Render, but it will still need some love from us. Don't worry, we'll have a look at it soon.
Title: Re: RAM issue when using Team Render
Post by: houska on 2019-01-07, 13:54:55
-Crash while exporting corona proxy when the object has more than one uv tag

loliolg, do you mean this issue?
https://forum.corona-renderer.com/index.php?topic=22212

That should be fixed already. I tried it just now and it doesn't crash for me. If you find a crashing situtation, we'd be happy to have the crashign object so that we could fix the crash for everyone. You can even use our private uploader so that the object is kept private.

Thanks!
Title: Re: RAM issue when using Team Render
Post by: houska on 2019-01-07, 14:45:40
-Corona proxy is not working with Team Render.

Do you have a scene or steps that would help us reproduce the issue? I created a simple scene, added a Corona Proxy object and it renders fine over TR both locally and on a remote render client.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-07, 15:11:26
-Corona proxy is not working with Team Render.

Do you have a scene or steps that would help us reproduce the issue? I created a simple scene, added a Corona Proxy object and it renders fine over TR both locally and on a remote render client.

I'm rendering another scene right now. But I will test it tomorrow and give you a feedback.
The main issue for me is just the thing with the RAM. Thats my favorite issue do be fixed. ;)
Title: Re: RAM issue when using Team Render
Post by: houska on 2019-01-07, 15:58:51
I'm rendering another scene right now. But I will test it tomorrow and give you a feedback.
The main issue for me is just the thing with the RAM. Thats my favorite issue do be fixed. ;)

Thanks and we of course understand that this issue is very unpleasant. We're already busy investigating it and will fix it in one of the upcoming daily builds!
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-08, 19:45:36
I wanted to test the problem I had with the Proxy. For that I updated to Corona 3 final and activated a demo license.
But now TeamRender isn't working anymore. (With the native render engine is everything working)
I got thousands of "[Corona4D]
Aborting synchronization: dumpExr failed" on my render client and I had to quit it with the task-manager. 

What could be the reason for that?
Title: Re: RAM issue when using Team Render
Post by: TomG on 2019-01-08, 19:49:00
Did you run the Corona Licensing Server? It will need to be running in order to hand out authorizations to the Client machines.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-08, 20:15:15
Did you run the Corona Licensing Server? It will need to be running in order to hand out authorizations to the Client machines.

No, I didn't. And to be honest, I can't even find an application called "Corona Licensing Server". Is it even an application?
Where should it be located on Mac? Thanks! :)
Title: Re: RAM issue when using Team Render
Post by: TomG on 2019-01-08, 20:27:50
It may not have been installed by default - I'd re-run the installer, choose Custom, and ensure that it's checked.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-08, 20:57:15
It may not have been installed by default - I'd re-run the installer, choose Custom, and ensure that it's checked.

Thanks!
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-09, 12:50:30
Corona Proxy isn't working here. I work on Mac and render on pc. Could it be a reason? I thought I have read something here in another topic.
Title: Re: RAM issue when using Team Render
Post by: TomG on 2019-01-09, 15:15:47
Do you mean Corona Proxy with Team Render, or Corona Proxy on its own? Sounds like it may be a separate issue from the original thread topic of "RAM issue when using Team Render" and may be best in a separate thread so it doesn't get lost :)
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-09, 16:36:31
Do you mean Corona Proxy with Team Render, or Corona Proxy on its own? Sounds like it may be a separate issue from the original thread topic of "RAM issue when using Team Render" and may be best in a separate thread so it doesn't get lost :)

I will make a separate and more detailed report later when I'm back at home.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-15, 12:33:00
I rendered an animation for around 15h last night and my RAM went up to 2.8GB only. Rendered with the latest Corona 3 version.
So maybe you already fixed this issue? But I can't see something new about that in the Corona 3 Changelog. Anyway well done and thank you ;)
Title: Re: RAM issue when using Team Render
Post by: HFPatzi on 2019-01-15, 15:35:15
Was it the first thing you rendered after opening cinema 4D? Sometimes the high RAM-Usage on my end occurs when starting another render without restarting cinema. Seems like the ram doesn't getr flushed after finishing a render.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-16, 23:28:56
Was it the first thing you rendered after opening cinema 4D? Sometimes the high RAM-Usage on my end occurs when starting another render without restarting cinema. Seems like the ram doesn't getr flushed after finishing a render.

Yes, I had this issue too. Thats why I always restart C4D bevor I render another big Image.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-17, 00:52:16
I just rendered an 5k image which took around 25min and my RAM went up to 25GB.
So I think it's less a matter of time and more of resolution.
Title: Re: RAM issue when using Team Render
Post by: lollolo on 2019-01-18, 20:30:42
Thank you for the fix. Have a nice weekend!
Title: Re: RAM issue when using Team Render
Post by: HFPatzi on 2019-01-22, 13:42:35
Seems like the RAM-issue is solved (4 x 9k-stills, one aftrer each other and no RAM-Warning).
Nevertheless i do experience some Sync-Errors with chunk-size at 50MB.
With the size at 50MB 2 of 11 clients get the error (Always different ones)
If i increase the chunk-size i.e. to 100MB even more clients get this error.
I then decreased the size to 12MB and had no errors.
The Sync-errors seem to always appear right after a new pass is beeing started.

In the console it looks like the following:
Code: [Select]
2019/01/22 11:44:32  [Corona4D] [TR] Sending HELLO to server
2019/01/22 11:44:32  [Corona4D] [TR] Connection established
2019/01/22 11:44:35  [Corona4D] REPORT: unchanged: 0, Prim: 46, Inst: 0, Mtl: 0, Env-tentative: 1
2019/01/22 11:44:35  [Corona4D] Core::renderFrame before onFrame
2019/01/22 11:44:35  [Corona4D] Parsing scene took 19.759 seconds
2019/01/22 11:44:35  [Corona4D] Calculating displacement
2019/01/22 11:44:35  [Corona4D] Preparing geometry took 0 seconds
2019/01/22 11:44:35  [Corona4D] Preparing geometry
2019/01/22 11:44:35  [Corona4D] Embree: memory usage: 232,8 MB. Total embree build time is 389ms, top level commit took: 301ms. There are 46 embree geometry groups, 0 instances, 46/0/0/0 tri/animTri/hair/custom primitives
2019/01/22 11:44:36  [Corona4D] Core::renderFrame after onFrame
2019/01/22 11:44:36  [Corona4D] CoronaCore::renderFrame before unique materials enum
2019/01/22 11:44:36  [Corona4D] CoronaCore::renderFrame after unique materials enum
2019/01/22 11:44:36  [Corona4D] CoronaCore::renderFrame after unique materials update
2019/01/22 11:44:36  [Corona4D] Preparing geometry took 0.57 seconds
2019/01/22 11:44:36  [Corona4D] Preparing lights
2019/01/22 11:44:36  [Corona4D] TreeAdapt built: nodes 23, time 0 ms, memory 5.224 kB
2019/01/22 11:44:36  [Corona4D] CutCache initialized: records 842580, time 12 ms, memory 33.7034 MB
2019/01/22 11:44:36  [Corona4D] CoronaCore:RenderFrame: after directLight
2019/01/22 11:44:36  [Corona4D] CoronaCore::renderFrame: after GI precompute
2019/01/22 11:44:36  [Corona4D] Preparing lights took 0.014 seconds
2019/01/22 11:44:36  [Corona4D] CoronaCore::renderFrame: before render
2019/01/22 11:44:36  [Corona4D] Rendering initial pass
2019/01/22 11:44:56  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 11:45:24  [Corona4D] [TR] Sending chunk 5/17 to the server
2019/01/22 11:45:51  [Corona4D] [TR] Sending chunk 6/17 to the server
2019/01/22 11:46:19  [Corona4D] [TR] Sending chunk 7/17 to the server
2019/01/22 11:46:45  [Corona4D] [TR] Sending chunk 8/17 to the server
2019/01/22 11:47:16  [Corona4D] [TR] Sending chunk 9/17 to the server
2019/01/22 11:47:45  [Corona4D] [TR] Sending chunk 10/17 to the server
2019/01/22 11:48:08  [Corona4D] [TR] Sending chunk 11/17 to the server
2019/01/22 11:48:15  [Corona4D] Rendering pass 2
2019/01/22 11:48:31  [Corona4D] Rendering initial pass
2019/01/22 11:48:31  [Corona4D] [TR] Sending chunk 12/17 to the server
2019/01/22 11:48:57  [Corona4D] [TR] Sending chunk 13/17 to the server
2019/01/22 11:49:20  [Corona4D] [TR] Sending chunk 14/17 to the server
2019/01/22 11:49:47  [Corona4D] [TR] Sending chunk 15/17 to the server
2019/01/22 11:50:13  [Corona4D] [TR] Sending chunk 16/17 to the server
2019/01/22 11:50:38  [Corona4D] [TR] Sending chunk 0/17 to the server
2019/01/22 11:51:03  [Corona4D] [TR] Sending chunk 1/17 to the server
2019/01/22 11:51:33  [Corona4D] [TR] Sending chunk 2/17 to the server
2019/01/22 11:51:48  [Corona4D] Rendering pass 2
2019/01/22 11:52:00  [Corona4D] Rendering initial pass
2019/01/22 11:52:00  [Corona4D] [TR] Sending chunk 3/17 to the server
2019/01/22 11:52:28  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 11:52:57  [Corona4D] [TR] Sending chunk 5/17 to the server
2019/01/22 11:53:23  [Corona4D] [TR] Sending chunk 6/17 to the server
2019/01/22 11:53:46  [Corona4D] [TR] Sending chunk 7/17 to the server
2019/01/22 11:54:12  [Corona4D] [TR] Sending chunk 8/17 to the server
2019/01/22 11:54:36  [Corona4D] [TR] Sending chunk 9/17 to the server
2019/01/22 11:55:04  [Corona4D] [TR] Sending chunk 10/17 to the server
2019/01/22 11:55:21  [Corona4D] Rendering pass 2
2019/01/22 11:55:28  [Corona4D] Rendering initial pass
2019/01/22 11:55:28  [Corona4D] [TR] Sending chunk 11/17 to the server
2019/01/22 11:55:53  [Corona4D] [TR] Sending chunk 12/17 to the server
2019/01/22 11:56:17  [Corona4D] [TR] Sending chunk 13/17 to the server
2019/01/22 11:56:45  [Corona4D] [TR] Sending chunk 14/17 to the server
2019/01/22 11:57:12  [Corona4D] [TR] Sending chunk 15/17 to the server
2019/01/22 11:57:36  [Corona4D] [TR] Sending chunk 16/17 to the server
2019/01/22 11:57:57  [Corona4D] [TR] Sending chunk 0/17 to the server
2019/01/22 11:58:28  [Corona4D] [TR] Sending chunk 1/17 to the server
2019/01/22 11:58:52  [Corona4D] [TR] Sending chunk 2/17 to the server
2019/01/22 11:58:59  [Corona4D] Rendering pass 2
2019/01/22 11:59:18  [Corona4D] Rendering initial pass
2019/01/22 11:59:18  [Corona4D] [TR] Sending chunk 3/17 to the server
2019/01/22 11:59:45  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:08  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:12  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:13  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:17  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:17  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:21  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:22  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:26  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:26  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:30  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:30  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:34  [Corona4D] [TR] Sending chunk 4/17 to the server
2019/01/22 12:00:34  [Corona4D] [TR] Frame synchronization failed: Communication Error
2019/01/22 12:00:35  [Corona4D] CoronaCore::renderFrame: after render
2019/01/22 12:00:35  [Corona4D] Terminating DR slaves
2019/01/22 12:00:35  [Corona4D]  - terminating slave handlers
2019/01/22 12:00:35  [Corona4D]  - waiting for broadcast thread to finish
2019/01/22 12:00:35  [Corona4D]  - clearing slave handlers
2019/01/22 12:00:35  [Corona4D] Terminating DR slaves ended
2019/01/22 12:00:35  [Corona4D] Rendering took 958.941 seconds
2019/01/22 12:00:35  [Corona4D] Cleaning up
2019/01/22 12:00:35  [Corona4D] Terminating DR slaves
2019/01/22 12:00:35  [Corona4D]  - terminating slave handlers
2019/01/22 12:00:35  [Corona4D]  - waiting for broadcast thread to finish
2019/01/22 12:00:35  [Corona4D]  - clearing slave handlers
2019/01/22 12:00:35  [Corona4D] Terminating DR slaves ended
2019/01/22 12:00:35  [Corona4D] CutCache memory: 34.9085 MB
2019/01/22 12:00:35  [Corona4D] Unique Primitives: 3251832
2019/01/22 12:00:35  [Corona4D] Primitives with instancing: 3251832
2019/01/22 12:00:35  [Corona4D] Area lights: 13
2019/01/22 12:00:35  [Corona4D] Geometry groups: 46
2019/01/22 12:00:35  [Corona4D] Materials: 23
2019/01/22 12:00:35  [Corona4D] Instances: 46
2019/01/22 12:00:35  [Corona4D] Portals: 0
2019/01/22 12:00:35  [Corona4D] Area lights: 13
2019/01/22 12:00:35  [Corona4D] Avg samples per pixel: 0.00199011
2019/01/22 12:00:35  [Corona4D] Avg rays per sample: 23.509
2019/01/22 12:00:35  [Corona4D] Rays/s: 2634.59
2019/01/22 12:00:35  [Corona4D] Samples/s: 112.067
2019/01/22 12:00:35  [Corona4D] Saving + Cleaning up took 0.041 seconds
2019/01/22 12:00:35  [Corona4D] CoronaCore::exiting renderFrame
2019/01/22 12:00:35  [Corona4D] Rendered 0/0 passes

It would be great if there will be a solution where we don't have to fiddle around with this values every time, since it's a bit time consuming

Anyways, thank you for your great product, that i don't want to miss anymore ;)

Greetings,
Moritz
Title: Re: RAM issue when using Team Render
Post by: Gr100 on 2019-02-22, 06:15:38
Happy I came here to know that this problem isn’t only happening to me. My computer maxes out right away no matter the resolution of the rendering. Power usage, CPU and RAM are through the roof. Computer Locks up and crashes. Sometimes the rendering doesn’t even start. A project that was done and ready to render out took me an additional weak just playing around to find ways to lower the file size etc. Files was only 35 MB.

Hi!

I experience almost the same issues with RAM usage in Teamrender. Except my RAM usage while rendering a high resolution still image with teamrender is almost instant at max. While rendering the same scene on my workstation only just uses around 3 to 4GB of RAM (see attached screenshots). Also nearly all my teamrender-clients crash while teamrendering this project. Unfortunatly without creating a bug report. The Teamrender client just quits.

I'm using the latest Corona Daily Build:

Full-speed non-debug version
Build timestamp: Oct 31 2018 17:15:05
Version: B2 (core 3)
Cinema version: CINEMA 4D Studio R18.057 S
Plugin status: initialized successfully


I have set a small render region in the cinema 4D render settings and set corona to render viewport selection only, singe i only needed this part of the image.

I'm sorry for not uploading the scene file because of nda. If you need any further informations, i'm happy to answer your questions.

Thanks for your Support!

Greetings from Germany,
Moritz
Title: Re: RAM issue when using Team Render
Post by: HFPatzi on 2019-02-22, 08:20:15
One thing you could try, but only to proof a theory: Try start a render with no multipasses, just the beauty pass. If you're using Lightmix, you could try to disable this. I had one pretty huge image (15k x 8k) where i had no problems rendering just the beaty pass, but with 5 additional mask passes on, my i got RAM-warnings.

Again, this is just to proof my theory here, as it should be possible to (Team)render images with multipasses and lightmix on and RAM-warnings off ;)

Also try to restart all client's and your workstation's cinema 4d before starting the render. Sometimes that helps.

Greetings,
Moritz