Long Parsing scene - 50% ram and 50% processor usage

[Jak]

Hello,

I am rendering a couple of animation sequences, the loading times for parsing scene is quite long, takes aproximately 2 - 3 minutes, with usage of half of my computers RAM, and oscilating between 50-60% of usage of processing power.

I read that parsing scenes takes time because of low RAM, however this is not the case here. Is there anything that I can do to remove parsing times? 2-3 minutes extra parsing times for each of the hundreds of frames is quite unfortunate.

EDIT: More information about my scene (as per information here: https://coronarenderer.freshdesk.com/support/solutions/articles/5000675854-system-is-running-low-on-ram-you-may-experience-excessive-parsing-times-and-or-crashes-you-will-pro)
[altough I know all of the points below are mainly to lower RAM usage, which is not necessary in my scene)

1) I have only 4 passes including Beauty pass; no atmospherics or AO pass - denoising 4 passes only.
2) No displacement
3) No super high resolution textures
4) Lots of instances and proxies
5) Resolution in HD only (not full HD)
6) Ram usage 50 percent, so I have enough RAM (47GB RAM usage, ~82GB RAM installed total)

Any info would be appreciated.

[rowmanns]

Hi,

Would you be able to send over a copy of your scene?

We will take a look to see if something can be optimised, either in your scene or in Corona.

Instructions on how to do this are in my signiture.

Thanks,

Rowan

[Jak]

Hi,

Would you be able to send over a copy of your scene?

We will take a look to see if something can be optimised, either in your scene or in Corona.

Instructions on how to do this are in my signiture.

Thanks,

Rowan

Well, this is still the same scene where I had problems saving any format with glare and bloom (posted problem in a thread earlier - doubled my RAM and I can save it now), so the file is still under NDA as it was before, which is why I am still unable to send the file for inspection.
Im curious if this has something to do with Corona scatter and proxies (as I am using Corona scatter and quite a lot of instances, plants, trees, cars etc).

I think I will power through this, but its a real shame, because I have one frame set to stop and save at the 10 minute mark, so every 500 frames I lose time for additional ~100/150 frames due to the precomputation ie parsing scene.

The scene is quite big might I add, I have managed to divide it into two chunks, however the precomputation is still there.

Once the precomputation is done, during the rendering the processor is at 100 percent usage (and at half RAM, same as precomputation), so there is no problem there. Its only during parsing scene Corona is not using full RAM (as per parsing scene instructions) or full processing power.

[Jak]

Hi,

(...)

We will take a look to see if something can be optimised, either in your scene or in Corona.

(...)

Rowan

The problem is getting worse. During mid render of an animation sequence, the precoputation has increased from 2 minutes now to ~7 minutes. At this rate I am no way going to make it on schedule with rendering the animation. How is it possible that the precomputation increased? The camera has not moved that far, nor did I add any new objects/materials to the scene.

@Rowan, you mentioned scene optimisation. Could you specify what optimisations you have in mind so that I can try and fix this issue myself? What else is there apart from adding proxies in order to cut down on memory as far scene optimisation goes? I hid geometry that is not being rendered during the camera movement, but cannot hide the objects that are going to be in view eventually. Can you please either link me to an article or help file, or could you state what I can do in this thread?

Also regarding Corona optimisation, what did you have in mind? What settings could help to lower the precomputation, per you post in terms of Corona optimisation? Could you either post a link to such information here, or post some of this information yourself?

EDIT: I am using 3dsmax and RAM usage is still at ~50% during the precomputation scene.

[Ink Visual]

Hi, I have similar problem with scene parsing rising over time as animation frames are being rendered. Last week it would grow from 3mins to almost 7mins during 48 hours of rendering.
The scene is indeed very heavy as it's a huge masterplan in mountainous environment but it was optimised in numerous ways already. Standard stuff, there are no big textures used, heavy polys are proxies, instances used wherever it was possible.
Last step that I've done yesterday that helped reducing parsing times was going through all the proxies once again, converting them back to mesh and running prooptimizer on them. On some sequences it reduced parsing from 3mins to 1.5mins, although the problem with parsing rising over time is still there.

[Vlad_the_rant]

Corona proxies DO NOT save RAM! Their only purpose is speeding up viewport performance, not RAM usage reduction. Therefore lots of proxies are, at best, the same as no proxies at all and at worst detrimental in very large scenes (since they all need to be loaded into RAM before rendering starts). If they are stored on a slow drive or on a network path, loading them will take time. Can you try removing temporarily at least the largest proxies or half the proxies to see whether that speeds up the pre-render preparation? Thanks!

[Jak]

Hi, I have similar problem with scene parsing rising over time as animation frames are being rendered. Last week it would grow from 3mins to almost 7mins during 48 hours of rendering.
The scene is indeed very heavy as it's a huge masterplan in mountainous environment but it was optimised in numerous ways already. Standard stuff, there are no big textures used, heavy polys are proxies, instances used wherever it was possible.
Last step that I've done yesterday that helped reducing parsing times was going through all the proxies once again, converting them back to mesh and running prooptimizer on them. On some sequences it reduced parsing from 3mins to 1.5mins, although the problem with parsing rising over time is still there.

Good to know its not only me that is having these issues.

Corona proxies DO NOT save RAM! Their only purpose is speeding up viewport performance, not RAM usage reduction. Therefore lots of proxies are, at best, the same as no proxies at all and at worst detrimental in very large scenes (since they all need to be loaded into RAM before rendering starts). If they are stored on a slow drive or on a network path, loading them will take time. Can you try removing temporarily at least the largest proxies or half the proxies to see whether that speeds up the pre-render preparation? Thanks!

Is blows my mind that for bigger scenes using corona proxies will actually make your scene less optimised and will take long to render... So it is better just to get a better graphics card set up and use as less or better yet no proxies at all. If its only for better viewport how does it differ from bounding box preview from within 3dsmax (apart from better overall visibility in terms of placement)? Will do some tests later for myself.

I am rendering another camera angle, started out with scene parsing 2 minutes, ater 3/4 of the animation done, the parsing increased to 5 minutes... Which is mind boggling, because its still the same aerial shot, the camera just rotated about 40 degrees, and no new geometry has been introduced to the scene, it is all there in that one shot from the beginning until the end. Its the same problem for all animation angles I render, the parsing time increases about 150% after half of the animation is done...

[romullus]

Is blows my mind that for bigger scenes using corona proxies will actually make your scene less optimised and will take long to render... So it is better just to get a better graphics card set up and use as less or better yet no proxies at all. If its only for better viewport how does it differ from bounding box preview (apart from better overall visibility in terms of placement)? Will do some tests later for myself.

Proxies also makes max file much lighter, which really helps with saving times, especially autosave.

[Jak]

Update: Set the scene to stop rendering after 5:30 minutes. It takes 14 minutes to complete one frame at this moment. So out of 14 minutes only 5:30 is actual render time. It got this bad after rendering 150 frames. Took about 7 or 8 minutes to render for the first 100 frames or so, including the 5:30 rendering time for one frame.

[arqrenderz]

Corona proxies DO NOT save RAM! Their only purpose is speeding up viewport performance, not RAM usage reduction. Therefore lots of proxies are, at best, the same as no proxies at all and at worst detrimental in very large scenes (since they all need to be loaded into RAM before rendering starts). If they are stored on a slow drive or on a network path, loading them will take time. Can you try removing temporarily at least the largest proxies or half the proxies to see whether that speeds up the pre-render preparation? Thanks!

Hi Papuca, we had such issue with a big scene, one thing that bothers us is the fact that Proxies take ram EVEN on a HIDDEN layer! WHY??? they are not being calculated for anything, so why they need to be loaded.... We were trying a script to change all the proxys geometry to a box but we ran in few bugs and had no more time to make it work, the script was working indeed, so if the corona team can do something about it, it would be really great for all.

[Jak]

Is blows my mind that for bigger scenes using corona proxies will actually make your scene less optimised and will take long to render... So it is better just to get a better graphics card set up and use as less or better yet no proxies at all. If its only for better viewport how does it differ from bounding box preview (apart from better overall visibility in terms of placement)? Will do some tests later for myself.

Proxies also makes max file much lighter, which really helps with saving times, especially autosave.

I would take lower rendering times and higher autosave time without a doubt. I am working on SSD at this moment and I still am having long parsing times.

Edit:

I found other people have had similar problems:
https://forum.corona-renderer.com/index.php?topic=24802.15

However I have 82 GB of ram and still having problems...

[Jak]

Update: Set the scene to stop rendering after 5:30 minutes. It takes 14 minutes to complete one frame at this moment. So out of 14 minutes only 5:30 is actual render time. It got this bad after rendering 150 frames. Took about 7 or 8 minutes to render for the first 100 frames or so, including the 5:30 rendering time for one frame.

Stopped rendering, restarted computer, loaded scene in and deleted all hidden layers

(...)one thing that bothers us is the fact that Proxies take ram EVEN on a HIDDEN layer! WHY???(...)

.
Started rendering animation, took 3 minutes to ready the frame (instead of ~9), and now its rendering... what the hell is going on...

Its as if Corona is somehow getting clogged up after a couple of hours of rendering, and its not my RAM either, because my ram usage is constanly at 50-60%. And its not 3dsmax either, as the person in the post I linked above is working under Cinema4D and still has these problems.

[romullus]

Hi Papuca, we had such issue with a big scene, one thing that bothers us is the fact that Proxies take ram EVEN on a HIDDEN layer! WHY??? they are not being calculated for anything, so why they need to be loaded....

Do you have keep in memory enabled on your proxies? Try to disable it and see if that solves the issue.

[Jak]

Another update:

Frame #1     : Total time to complete this frame ~7:30 minutes (5:30 actual render time, rest for parsing and preparing geometry)
Frame #88 : Total time to complete this frame ~10:37 minutes (5:30 actual render time, rest for parsing and preparing geometry)

Each frame is longer than the previous frame by seconds, and this piles up towards the later frames. And again, this does not make sense, as its the same geometry from an aerial view, only the camera is slighly rotated (yes slightly, as it is a slow shot).

Total estimated time was 26 hours for the completion of my animation. Now total estimated time is 32 hours (after 14 hours have passed). Still have 120 frames to render. Im really curious what the rendering time will be in another 80 frames.

How can anyone use Corona for production in animation at this point?

Also is there anyone from Corona support here?

@rowmanns, you mentioned scene optimisation, so I ask one more time. How can I optimise my scene to help solve this issue? If you have an idea of how to solve this can you please post info in this thread? Also, someone already had sent you a file with very similar problems (C4D file), did Corona team have any chance to look into the problem? Thanks.

[maru]

Papuca is from our support team, and Rowmanns from our QA teams. We are investigating this.
If you need immediate assistance, it is always best to use support@corona-renderer.com or https://coronarenderer.freshdesk.com/support/tickets/new

[rowmanns]

Hi,

I can understand the frustration here, but if we can't reproduce the issue it is really hard for us to fix it. Of the top of my head I don't have any ideas for optimisation, which is why a copy of the scene for max would be most helpful for us to identify this issue.

We can treat your scene confidentially if needed. For the most immediate response I'd recommend contacting the support channel as suggested by Maru and we'll do our best to find some solution.

Thanks,

Rowan

[Jak]

Hi,

I can understand the frustration here, but if we can't reproduce the issue it is really hard for us to fix it. Of the top of my head I don't have any ideas for optimisation, which is why a copy of the scene for max would be most helpful for us to identify this issue.

We can treat your scene confidentially if needed. For the most immediate response I'd recommend contacting the support channel as suggested by Maru and we'll do our best to find some solution.

Thanks,

Rowan

This is actually going beyond frustration...

Once again, I have an NDA signed, as well as I dont have time to waste to prepare my scene in order to send it to you guys and wait for a fix. I have a deadline on my shoulders and this came as quite the surprise as you can imagine.

I am rendering frame 103, and it is taking more than 25 minutes (!) to prepare the geometry now. First frame 2 and a half minutes, now its taking 25 minutes a 100 frames later! I had to force close the animation and will restart the computer now to see if I can continue from this point and get the 2.5 minutes like I had at the begining.

At this rate I will never finish this animation.

Not sure what to do now, but one thing is certain. Until this gets fixed I either cannot do animations, or have to switch my rendering engine.


Here is a screen dump as proof, so that people dont think I dont have better things to do than make things up:

[Jak]

After restart:

~2:00 parsing + preparing geometry (please find attachment) (one frame set to stop after 5:30 min mark).

Rendering 2 frames earlier from the moment I forcefully closed max and restarted the system.

RAM still @ ~60% as it is during rendering of every animation sequence of my current project.

[Jak]

Another update. The same frame that took 25 minutes (before force quit) before restart is now taking about ~2:30 minutes to get through preparing geometry:

[maru]

I think this thread is missing some vital information:
- Which exact version of Corona is used?
- Which Max version?
- Which Windows version? (it looks like Win 7)
- Are there some 3rd party plugins used? (iToo, Laubwerk, Siger, ......)
- Did you send your scene to us? Which upload method did you use, and what's the file name?
- Are any assets stored on the network? If so, does the issue persist if they are stored locally?

Some things that come to my mind based on the information we have right now:
1) It could be somehow related to textures - have you tried converting ALL textures in your scene to 3ds Max Bitmap or to Corona Bitmap? You can do this with the converter script (right click in any viewport > corona converter > then there are two options "Bitmap to CBitmap" and "CBitmap to Bitmap").
2) Proxies - what are your proxy settings? Are you using the "Store in RAM" option? Have you tried toggling it on/off?
3) Are you using the UHD Cache? (performance tab) If so, have you tried switching to path tracing only? (path tracing as both primary and secondary GI solver)
4) It could be some 3rd party app - 3ds Max, some other plugin, or antivirus/firewall doing some stuff in the background. This would be hard to diagnose.

To understand a problem and provide reliable solutions, we need to have as much information as possible, and it is best if we have the scene itself to reproduce that problem. Without it, it is really hard to help, and we can mostly blindly troubleshoot.

Also, did you start a support ticket? (by sending an email to support@corona-renderer.com or using this form https://coronarenderer.freshdesk.com/support/tickets/new)
I am asking, because I could not find it.

[Jak]

I think this thread is missing some vital information:
- Which exact version of Corona is used?
- Which Max version?
- Which Windows version? (it looks like Win 7)
- Are there some 3rd party plugins used? (iToo, Laubwerk, Siger, ......)
- Did you send your scene to us? Which upload method did you use, and what's the file name?
- Are any assets stored on the network? If so, does the issue persist if they are stored locally?

Some things that come to my mind based on the information we have right now:
1) It could be somehow related to textures - have you tried converting ALL textures in your scene to 3ds Max Bitmap or to Corona Bitmap? You can do this with the converter script (right click in any viewport > corona converter > then there are two options "Bitmap to CBitmap" and "CBitmap to Bitmap").
2) Proxies - what are your proxy settings? Are you using the "Store in RAM" option? Have you tried toggling it on/off?
3) Are you using the UHD Cache? (performance tab) If so, have you tried switching to path tracing only? (path tracing as both primary and secondary GI solver)
4) It could be some 3rd party app - 3ds Max, some other plugin, or antivirus/firewall doing some stuff in the background. This would be hard to diagnose.

To understand a problem and provide reliable solutions, we need to have as much information as possible, and it is best if we have the scene itself to reproduce that problem. Without it, it is really hard to help, and we can mostly blindly troubleshoot.

Also, did you start a support ticket? (by sending an email to support@corona-renderer.com or using this form https://coronarenderer.freshdesk.com/support/tickets/new)
I am asking, because I could not find it.

Seems I had missed this thread somehow.

Specs:

Win7 SP1, 3dsmax 2016, Corona version (previous stable build to Corona 5 - no daily builds), no third party plugins (other than bercon maps), UHD Cache (no path tracing if I remember correctly now).

I was using internal 3dsmax bitmap for pretty much most of the models, did not try switching to corona bitmap. The proxy store in RAM was unticked if I remember correctly now.

I was using libraries stored on my external SSD drive (connected through USB 2 not USB 3), which might have been a culrpit from what I have read.

No ticket, did not have time at that moment, now Im working on other works (stills), so dont think will make a ticket, since I dont have time ATM to try and make a similar scene with no NDA.

I had to extend deadline to finish the animation with increasing rendering times for each frames unfortunately but the job got done in the end.

Thanks for response, and sorry for my delayed reply.

[gabor]

Hi there,
I have the same problem. Constantly increasing scene parsing and preparing geomery times. Heavy scene, lots of proxies, geomerty and forrest pack objects (large grass areas). I realized this problem only after I started using Forrest pack.
Did you find a solution?

[maru]

Hi there,
I have the same problem. Constantly increasing scene parsing and preparing geomery times. Heavy scene, lots of proxies, geomerty and forrest pack objects (large grass areas). I realized this problem only after I started using Forrest pack.
Did you find a solution?

Are you using the newest versions of all software?
Can you send some more info? (such as those versions, and the scene itself if possible)

[gabor]

Newest Corona, newest Forrest Pack and max 2019. Unfortunately I can't archive the max file, it gives me an error code of 11, altough all the assets are available. I'm rendering this on 2 computers. My workstation is an intel i9-9900k, and the render slave is an amd TR2970wx. The inceasing "preparing geometry" is not the same. The i9 gets this much faster done. It starts with 0:14 seconds and with each frame it takes ca. 1 seconds more - which is acceptabe, since the camera is getting closer to the objects.
But on the threadripper it gets at least 10 seconds longer with each frame which is unacceptable in a one minute animation project. It starts with 0:21, then 0:42, then 0:53 and so on... I know, the threadripper needs more time for render preparations, since its not so fast on a silgle thread. I'm rendering with it since two weeks, and the rendertimes did not increased from one frame to the other this fast.
I've rendered this scene before, without Forrest Pack grass, and did not realize this problem.

[gabor]

Update: preparing geometry tales from one frame to the next longer and longer. But when I stop the rendering for example at frame 30 (where prep. geom. was 1:10 long) and I restart the rendering from frame 30, the prep. geom. needs again only 15 seconds.

[maru]

I am afraid we may not be able to help without having the problematic scene and trying to reproduce the problem on our side. If you can upload your scene, please do so (uploader is in my signature).
Other than that, I can recommend checking the various hints listed in this forum thread.
If you are using DR, you can also try the "Restart Max" option in the DR server, however this will add extra time for closing and re-starting Max, so I am not sure if it will result in a speedup at all.

[gabor]

I've opened a ticket, and uploaded the archive via dropbox.
I hope you can find a solution or a workaround, because with constantly growing rendertimes I can't keep the deadlines.
Thanks!

[mcnamex]

any news on this issue... I am having a similar issue here as well

[GeorgeK]

any news on this issue... I am having a similar issue here as well

Hi mcnamex, are you using forest pack or back-burner for batch rendering?

[maru]

(Internal ID=438429845)

[mcnamex]

any news on this issue... I am having a similar issue here as well

Hi mcnamex, are you using forest pack or back-burner for batch rendering?

Yeah FP,RC,Backburner. But rendering locally behaving the same as well.

[GeorgeK]

any news on this issue... I am having a similar issue here as well

Hi mcnamex, are you using forest pack or back-burner for batch rendering?

Yeah FP,RC,Backburner. But rendering locally behaving the same as well.

We are currently investigating this issue, if you could raise a ticket or  provide a scene for us it would be helpful.

[rowmanns]

Hi,

So thanks to Gabor for sending over a scene!

We are now able to reproduce the problem of the geometry processing times increasing with each frame in the animation.

I have fowarded this on to our dev team for futher investigation. I will let you know when I have some more information.

Cheers,

Rowan

[kv3t1n4c]

Hi guys,

I'm experiencing the same issue, using only cproxies and cscatter and some plugins like bercon maps, floor generator. All scattered instances are cproxies with enabled.
Look the attachements.
The biggest difference is not visible in these numbers - it's tansforming vertices - from several seconds in the begining rised to cca 7:30 minutes on 20th frame.
I will be able to send the scene on this friday probably. The animation is not crucial for me, I'm doing mostly stills, this was just a preview for possible output.

Adding some info:

win10
max 2019
corona5
amd threadripper 1950
64gb DDR4
nvidia rtx 2070

[kv3t1n4c]

I have another question. Is it beneficial as to performance to enable temporal consistency for this type of scattering? According to thumbnail I would say yes, but it apparently didn't afect this problem of scene parsing time getting longer.

Thanks ;)

[kv3t1n4c]

It seems there is something bad with max itself. Max became really tough, even when autosave file is saved within few seconds (160MB), max is unusable for several minutes (cca 5) with only one proc core fully loaded (no disk usage or rising memory consumption). I guess it also has something to do with many individual cscatter objects, in this way my scene is pretty heavy (26 csatters).

[GeorgeK]

Hi guys,

I'm experiencing the same issue, using only cproxies and cscatter and some plugins like bercon maps, floor generator. All scattered instances are cproxies with enabled.
Look the attachements.
The biggest difference is not visible in these numbers - it's tansforming vertices - from several seconds in the begining rised to cca 7:30 minutes on 20th frame.
I will be able to send the scene on this friday probably. The animation is not crucial for me, I'm doing mostly stills, this was just a preview for possible output.

Adding some info:

win10
max 2019
corona5
amd threadripper 1950
64gb DDR4
nvidia rtx 2070

Please ping back here when you have the archive uploaded, thank you.

[gabor]

Hey guys,
any news here? My support ticket was closed, but unfortunately there was no solution for the problem. Maybe this could be corrected in the upcoming version of Corona?
Did any of you find a possible workaround?

[mcnamex]

Hey guys,
any news here? My support ticket was closed, but unfortunately there was no solution for the problem. Maybe this could be corrected in the upcoming version of Corona?
Did any of you find a possible workaround?

Yeah would be interested in the solutions discovered so far

[rowmanns]

Hi,

I have a short update for this. We did some investigation and managed to find the root cause of this.

We have managed to fix the increasing times in scene parsing, this will be in the next v7 Daily Build.
BUT the increasing time in Geometry processing is still there. This is due to the fact that there is a bug in an external library we use which causes this. We are working with the devs of this library to fix this.

Thanks for your patience.

Rowan

[DustinMoore]

Any news on this issue? I have the same problem but parsing freezes Max and I have to force close every time. Unusable. Just downloaded the daily to see if it helps.

[rowmanns]

Hi,

Please submit a new thread explaining your issue. This one is pretty old and maybe have a different route cause.

Can you provide also the following information;
Corona version
3ds Max version
What exactly you were doing at the time
If possible your 3ds max scene
If possible a minidump from the freeze.

Instructions on how to send the files to us are in my signiture.

Cheers,

Rowan

[thomaslivings]

Hi,
I have the same problem out lined here. I have found a couple of threads about this frame time increasing problem but they offer no solutions. Has anyone resolved this problem of the parsing time increasing every frame?

[Avi]

Hello,

I am rendering a couple of animation sequences, the loading times for parsing scene is quite long, takes aproximately 2 - 3 minutes, with usage of half of my computers RAM, and oscilating between 50-60% of usage of processing power.

I read that parsing scenes takes time because of low RAM, however this is not the case here. Is there anything that I can do to remove parsing times? 2-3 minutes extra parsing times for each of the hundreds of frames is quite unfortunate.

EDIT: More information about my scene (as per information here: https://coronarenderer.freshdesk.com/support/solutions/articles/5000675854-system-is-running-low-on-ram-you-may-experience-excessive-parsing-times-and-or-crashes-you-will-pro)
[altough I know all of the points below are mainly to lower RAM usage, which is not necessary in my scene)

1) I have only 4 passes including Beauty pass; no atmospherics or AO pass - denoising 4 passes only.
2) No displacement
3) No super high resolution textures
4) Lots of instances and proxies
5) Resolution in HD only (not full HD)
6) Ram usage 50 percent, so I have enough RAM (47GB RAM usage, ~82GB RAM installed total)

Any info would be appreciated.

Hi,

I am from the corona team & I would need some additional info to investigate your scene.

1. Can you let me know the 3dsmax & corona version you are using?
2. Can you send me your scene for a close inspection. You can use our private uploaded to send us the file. Link : https://corona-renderer.com/upload

Once you are done uploading, do let me know the file name.

3. Can you let me know the step by step process to reproduce the same results?

Looking forward to hearing back from you soon.

Regards,
Avi

[gabor]

Any progress here? I am using the latest Corona with the latest 3ds max, still this same, very old problem exist. The more complex the scene is, rendertimes increasing more and more from frame to frame. Very annoying. Parsing scene and preparing geometry times are constantly increasing when rendering animations.

[Ink Visual]

Rewriting scene parsing has been on Trello board in Pool of Big Ideas for last couple versions now (I think dating back to 6 or 7 as far as I remember) but unfortunatelly it never made it to the developement stage? For rendering animations e mostly moved to the render farm so this problem doesn't affect us a lot anymore (farm is using hundreads of nodes each rendering only couple frames), but it was definately annoing to see the rendertimes grow frame after frame while we were rendering locally.