Chaos Corona Forum

Chaos Corona for 3ds Max => [Max] Bug Reporting => Topic started by: Dan Rodgers on 2019-08-02, 04:27:07

Title: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-08-02, 04:27:07
Hi,

We have an issue where in some scenes, and apparently for no reason, Corona starts to save out HUGE layered exr files rather than normal single image files.

The scenes haven't really changed, a couple of geometry changes, but all render settings are the same.  We are saving as half float exr.

Does Corona even support saving as multilayered exr?  I have never seen an option for it.


This is a massive issue for us, it takes 30 mins to save out the files and because it is doing this for every render element we can easily chew through a couple of hundred Gb of space overnight.

We are using Backburner, but we have the exact same issue if we manually save straight from the VFB


Cheers
Title: Re: Corona saving layered EXR on render end
Post by: jms.lwly on 2019-08-02, 10:29:30
I’ve had a similar (unexplained) issue recently - rendered on two machines via backburner, one saved out multilayered single EXRs, the other saved each channel as a separate file.

Literally the same render job from Backburner, but this happened consistently over several different camera passes.
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-08-02, 10:53:54
the other saved each channel as a separate file.

The other saved additionally each channel, right? The main output file still contained all other passes? It's a very strange behaviour. Sometimes I had the feeling that a node from which the job was submitted writes the (layererd) EXRs correctly (and only this) when included in the BB job. but I'm not sure. That mess in the output directories is really annoying :-|


Good Luck




Title: Re: Corona saving layered EXR on render end
Post by: alexyork on 2019-08-02, 14:52:29
Same here... happened once or twice. Unfortunately no time to diagnose.
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-08-02, 15:16:12
no time to diagnose.

Don't waste your time, because I fear you will find nothing. This is -again- a "since ever" issue with max :(

Does Corona even support saving as multilayered exr?  I have never seen an option for it.

If you use EXR as output and choose the "Setup" button you can activate "Automatically Add/Remove Render Elements from Render Dialog". This would put all your render elements into one EXR (Max feature, not a Corona feature). In pratice it happens often that additionally the render elements also get saved as extra files. And CXR (which is basically multilayered EXR + additional Corona specific information) has it all inclusive anyway :)


Good Luck



Title: Re: Corona saving layered EXR on render end
Post by: alexyork on 2019-08-02, 16:11:37
Are you guys using Deadline always? Maybe related?
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-08-12, 06:11:56
Cheers for all the info.  Never realised there was a native layered option within the 3ds max setup dialogue. It makes me surer that this is probably just a 3ds max issue, no wonder VRay had their own save options from years ago..

We are only using Backburner here.

We constantly get this issue though (see attached), where the files aren't even layered but are all huge and identical sizes except the beauty.  I can't split them with exr-io or in Fusion/AE  and it gobbles up server space on a crazy level.  Only seems to do it on large renders, 5k and above
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-08-12, 09:07:31
Oh, this looks like you are not using any compression. Go to "setup" while you are defining the output and use zip compression or wavelet (yes, the latter is also lossless, no worries).


Good Luck



Title: Re: Corona saving layered EXR on render end
Post by: maru on 2019-08-14, 12:37:24
Is the below correct?

- You would like to save only a single EXR file with all render elements included in it
- Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

---

One extra note:
If you use the "Save CXR" option and then convert your CXR to EXR, you will end up with a large file containing some layers which are normally hidden (for example information used to perform denoising). Maybe you are getting this kind of files? Could you send us some samples? The uploader is in my signature.
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-08-14, 19:04:22
Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

I guess it's again a Max issue not passing the file type properties (does not depend on output type EXR or CXR, even not on the renderer). We noticed that the node from which the job was submitted does it correctly if included in the BB job. Meaning: it saves only that one multilayered exr in this instance just like intended. All others do whatever. Completely weird, but also there "forever" in Max. Additionally I guess it's about the local settings of a node, just like JPG quality (had this issue recently while scripting).

Just tried it: seems to be as guessed :)

Use attached script as prerender script for a BB job (Render Setup -> Scripts -> Pre-Prender) and look at the max.log of the participating nodes: on nodes which write all additional passes you get 'false'. On nodes which work as intended you get 'true'. The script also sets this value to 'true' so that if you render ONE job with the script as prerender script on all nodes, the issue should be solved because the setting is locally stored on the node. Time to add this as another prerender script network wide.


Good Luck


Edit: This affects EXR only unfortunately, but not CXR.
Edit: Removed the script to avoid data loss. Reason: When using it (permanently as prerender script) and not wanting one layered EXR as output but multiple ones, additional REs are not saved at all. We always do layered EXRs so it does what it should do for us.
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-08-23, 03:34:03
Oh, this looks like you are not using any compression. Go to "setup" while you are defining the output and use zip compression or wavelet (yes, the latter is also lossless, no worries).


Good Luck

That's how it is set up anyway, with ZIP compression.

Is the below correct?

- You would like to save only a single EXR file with all render elements included in it
- Right now, instead of a single file, you are getting the "main" EXR + all other render elements saved separately as EXR files

---

One extra note:
If you use the "Save CXR" option and then convert your CXR to EXR, you will end up with a large file containing some layers which are normally hidden (for example information used to perform denoising). Maybe you are getting this kind of files? Could you send us some samples? The uploader is in my signature.

No.  I want all my exr's saved out separate (which is happening).  The issue is that the files are bigger than they should be. MUCH bigger. Even if I make a mistake and have an empty render element i.e Volumetrics but I have no volumes in the scene, it will save out a massive file with no info in it.  Check the image above, there is no way that my alpha should be more than double the size of my beauty pass, and it has no extra info in there that I can access.



So after more testing, this isn't just an issue with Backburner...  and we dont get the issue with VRay, Redshift etc.  It seems to happen with only the high res files.  3k, we get the expected different file sizes per pass, jump to 6k or 7k and we get all [asses with the same file size.  This also happens with TIFFs.  And across every machine in the studio.


I genuinely think this is a Corona issue, something about saving large-resolution files seems to make it shit it's pants, the contents of which are filling up our server space
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-08-23, 09:45:53
Hi,

Does this only happen when using backburner? Or does it also happen when rendering with Corona on its own?

Secondly, would you be able to provide a scene where this is happening for you? And I will do my best to investigate the issue.

Instructions on how to send the scene to us are in my signiture.

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: valwizard on 2019-08-25, 13:53:27
It is happening when using DR. I have created a topic but perhaps it was deleted in order to not duplicate the matter
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-09-02, 08:08:36
Hi,

Does this only happen when using backburner? Or does it also happen when rendering with Corona on its own?

Secondly, would you be able to provide a scene where this is happening for you? And I will do my best to investigate the issue.

Instructions on how to send the scene to us are in my signiture.

Thanks,

Rowan

this happens with both.  The only way to get 'normal sized' render passes is to save as CXR then save out p[asses individually from that.

I'll try and get a file over to you by then end of the week
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-09-02, 16:30:02
Thanks, I'll look forward to receiving the scene.
Title: Re: Corona saving layered EXR on render end
Post by: denisgo22 on 2019-09-02, 19:19:23
Yes this always max problem/and this always happened with corona render  and whether to use backburner or not, because what file extension will be in Common/render output of 3dmax,
the same will always be in Render Elements:)
But Vray have separate from max frame buffer options, that override max options for saving files , that allow to saving output files in different formats or not saving at all separately render
channels  //, for example channels in jpg and main max output to exr.
which, to a great inconvenience, not exist in Corona render:((
time to pay attention to this in a future release:))
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-09-11, 15:39:44
That's how it is set up anyway, with ZIP compression.

In your original post, you wrote, the huge(r) files are layered. What makes you think so? I still bet the render elements you've shown in the screenshot are just uncompressed. Max seems to remember the exr settings at creation time of a render element. You can reproduce this doing the following (no DR or BB involved and independently from render size - even works like this using scanline renderer):



-> WireColor is uncompressed, Cmasking_Mask is compressed.

So what you've shown in the screenshot here (https://forum.corona-renderer.com/index.php?topic=25686.msg154107#msg154107) could be the result of setting up the render elements on a node/at a time where exr compression was switched off for exr output and then submitting/rendering it on another node with other exr settings; compression active that is.

All in all I think there are at least three topics mixed in this thread meanwhile? To sort it out:



As for the latter: It seems that even if you do not explicitly specify a file name/type for a render element (usually a bad idea), the exr settings at creation time get stored and are not adopted from the current main output settings. At least compression settings - have not tested others. Well, that's new to me :)



Good Luck




Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-09-12, 01:34:09
That's how it is set up anyway, with ZIP compression.

In your original post, you wrote, the huge(r) files are layered. What makes you think so?


I opened with EXR.IO plugin in photoshop and each file was layered.  But tbh that looks like it was a very rare glitch, we haven't had the issue many times, and possibly could have been an issue with max.  I should have probably started a new thread about the other issue of exr file sizes all being the same...



So what you've shown in the screenshot here (https://forum.corona-renderer.com/index.php?topic=25686.msg154107#msg154107) could be the result of setting up the render elements on a node/at a time where exr compression was switched off for exr output and then submitting/rendering it on another node with other exr settings; compression active that is.

All in all I think there are at least three topics mixed in this thread meanwhile? To sort it out:

  • Original issue with large render element files (I suspect just uncompressed EXRs, see above)
  • Issue having a layered EXR but also additionally all passes as extra EXR (that's how I misinterpreted it, cause: local settings)
  • The dependency between main output file type and render elements file type(s) as denisgo22 mentioned


As for the latter: It seems that even if you do not explicitly specify a file name/type for a render element (usually a bad idea), the exr settings at creation time get stored and are not adopted from the current main output settings. At least compression settings - have not tested others. Well, that's new to me :)


I genuinely think this is a Corona issue.  In our studio, we do colour drafts at 3.5k and then final images at 6k or 7k.  The 3.5k renders are always fine and render elements have different file sizes, as you would expect, they are saved out as EXR.  The settings don't change, we change the output size to 6k/7k, render, and then we have the issue that I screenshotted, where each file is large and the exact same size.  We regularly use VRay, Redshift and sometimes FStorm in the studio and never get this issue, and we never had it before Corona 4.  Everything is setup exactly the same I we have done for the past 3-4 years.

I will try some more tests as you have mentioned though.




Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-10-14, 11:08:31
Hi,

We will investigate this.

Thanks,

Rowan

(Internal ID=409573480)
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-10-31, 02:30:16
any news on this yet?

We are getting to a point where we have to keep clearing old renders out on a regular basis because they are eating up so much server space.  We have the same problem with TIFF files too.

Our only solution seems to be save as CXR, save out passes manually then delete cxr.
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-10-31, 09:47:23
Hi Dan

We are looking into this, but so far we have been unable to reproduce this behaviour yet.

Do you have any further reproduction steps which may help us?

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2019-10-31, 13:14:35
we have been unable to reproduce this behaviour yet.

What exactly have you tried? Remember that you have to send the scene through Backburner/farm (another node) to at least get the effect of layered EXR+extra EXR for every render element (RE) in some cases.

Here are questions for you:

1. when using CXR as output and having additonal REs defined. Should there be a) only a single, multilayered CRX as output or b) the multilayered CXR plus additional CXRs for every RE? If b), what is it for?

2. when using EXR as output and having additional REs defined AND you have ticked "Automatically Add/Remove Render Elements" in Render Output File -> "Setup". What do you get when a) rendering locally and b) sending it through Backburner? Please use the attached script as prerender script in Render setup -> common -> scripts to log the current state of the EXR output settings on the node(s) into max.log. My findings are: it depends on local settings (that's what the script reports, don't worry it does not change anything). This should never be the case.


Good Luck


Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-10-31, 14:22:48
we have been unable to reproduce this behaviour yet.

What exactly have you tried? Remember that you have to send the scene through Backburner/farm (another node) to at least get the effect of layered EXR+extra EXR for every render element (RE) in some cases.

Here are questions for you:

1. when using CXR as output and having additonal REs defined. Should there be a) only a single, multilayered CRX as output or b) the multilayered CXR plus additional CXRs for every RE? If b), what is it for?

2. when using EXR as output and having additional REs defined AND you have ticked "Automatically Add/Remove Render Elements" in Render Output File -> "Setup". What do you get when a) rendering locally and b) sending it through Backburner? Please use the attached script as prerender script in Render setup -> common -> scripts to log the current state of the EXR output settings on the node(s) into max.log. My findings are: it depends on local settings (that's what the script reports, don't worry it does not change anything). This should never be the case.


Good Luck

I have done some testing this week. And for far it seems to save out correctly. i.e. I have one layered exr and then a seperate exr for each element which are the correct sizes.

Quote
1. when using CXR as output and having additonal REs defined. Should there be a) only a single, multilayered CRX as output or b) the multilayered CXR plus additional CXRs for every RE? If b), what is it for?

The answer to this is b). I get a multilayered cxr + an additional element for each render element. Honestly I don't know why it is like this and there is already a feature request logged to change this behaviour.

(Internal ID=403268985)

Quote
2. when using EXR as output and having additional REs defined AND you have ticked "Automatically Add/Remove Render Elements" in Render Output File -> "Setup". What do you get when a) rendering locally and b) sending it through Backburner? Please use the attached script as prerender script in Render setup -> common -> scripts to log the current state of the EXR output settings on the node(s) into max.log. My findings are: it depends on local settings (that's what the script reports, don't worry it does not change anything). This should never be the case.

I will get back to you on this after I have had the chance to do some more testing.

Cheers.
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-11-01, 02:22:35
Hi Dan

We are looking into this, but so far we have been unable to reproduce this behaviour yet.

Do you have any further reproduction steps which may help us?

Thanks,

Rowan

Well, that's annoying.  I have no idea how you're going to reproduce the issue then.

We just submit the job as normal to BB and the render is spat out a while later, there's no script or special process that we follow, we just use max's native functions.
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-11-01, 10:40:35
Well, that's annoying.  I have no idea how you're going to reproduce the issue then.

We just submit the job as normal to BB and the render is spat out a while later, there's no script or special process that we follow, we just use max's native functions.

Hi,

Which versions of Max and backburner are you using? I couldn't see this info in the thread..

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-11-06, 01:16:44
Well, that's annoying.  I have no idea how you're going to reproduce the issue then.

We just submit the job as normal to BB and the render is spat out a while later, there's no script or special process that we follow, we just use max's native functions.

Hi,

Which versions of Max and backburner are you using? I couldn't see this info in the thread..


Rowan

Max 2018.4

Just spotted our backburner manager hasn't been updated from 2016 to 2018.  I'm going to update now and I'll let you know if that fixes it
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-11-06, 03:32:38
didn't fix it..   

so we are not running max 2018.4 and BB 2018.
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-11-06, 10:53:57
Hi Dan,

Thanks for the info. I had been using 2019. I will install the exact versions you are using and see if I can reproduce it.

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2019-11-07, 13:33:31
Hi Dan,

Can you send me a screenshot of your file save settings for exr?

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: Dan Rodgers on 2019-11-14, 03:15:24
Hi Dan,

Can you send me a screenshot of your file save settings for exr?

Thanks,

Rowan

They should just be default, the only thing I change is every now and then I use full-float
Title: Re: Corona saving layered EXR on render end
Post by: maru on 2020-01-22, 13:52:50
I did manage to reproduce this, but only under very specific circumstances:
- You must be saving from the VFB> Save All (it does not happen when the files are automatically saved on render end)
- You must manually add all render elements to the list in the EXR save dialog
What happens then is:
- The beauty is saved in EXR format with all render elements stored inside it (can be found for example in Fusion, but not in PS)
- Each single render element is also saved in EXR format with all render elements stored inside it (again this can be found in Fusion)

So if the same behavior can be triggered when rendering via BB/DR, then I suppose this would be the culprit.
We will look into it further, and will see if the same happens with other renderers. If it only happens when using Corona's "Save all", then it would be a Corona-only issue. I can't test it right now.
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2020-01-23, 13:16:19
Maru, I tested this with BB2019 and Max2020 finally. I used a simple scene with a single render element added in the RE tab, not touched afterwards and not manually added to the EXR output. Maybe the case you described is just a subset:



1. Scanline and EXR:

BB job A: Autoadd ON in output configuration when submitting the scene:
   1) BB server local setting autoadd on: 1 layered EXR written (beauty + RE)
   2) BB server local setting autoadd off: 1 layered EXR written (beauty + RE) plus 1 flat EXR (the render element)

BB job B: Autoadd OFF in output configuration when submitting the scene:
   1) BB server local setting autoadd on: 1 flat EXR (beauty only), NO render element at all
   2) BB server local setting autoadd off: 1 flat EXR written (beauty only) plus 1 flat EXR for the render element

---------------------

2. Scanline and CXR:

   4 times 1 flat CXR (Beauty only) plus 1 flat CXR (the render element). No Alpha.

---------------------

3. Corona and CXR:

   4 times 1 layered CXR written (Beauty + RE) plus 1 flat CXR (the render element)

---------------------

4. Corona and EXR:

BB job A: Autoadd ON in output configuration when submitting the scene:
   1) BB server local setting autoadd on: 1 layered EXR written (beauty + RE)
   2) BB server local setting autoadd off: 1 layered EXR written (beauty + RE) plus 1 flat EXR (the render element)

BB job B: Autoadd OFF in output configuration when submitting the scene:
   1) BB server local setting autoadd on:1 flat EXR (beauty only), NO render elements at all
   2) BB server local setting autoadd off:1 flat EXR written (beauty only) plus 1 flat EXR for the render element




You can see that Corona + EXR is exactly the same as Scanline + EXR, a weird Max issue - not bound to a specific renderer. And again you can also see the impact of local settings. Case B.1 is the reason for removing the script from here. (https://forum.corona-renderer.com/index.php?topic=25686.msg154347#msg154347)That post is still valid for me, since I was able to reproduce it with the newest versions available. Only B.2 and A.1 are correct results, B.1 is data loss in the worst case and A.2 creates annoying additional render elements as Corona CXR always does ;)


Good Luck




Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2020-02-03, 18:09:53
You can see that Corona + EXR is exactly the same as Scanline + EXR, a weird Max issue - not bound to a specific renderer. And again you can also see the impact of local settings. Case B.1 is the reason for removing the script from here. (https://forum.corona-renderer.com/index.php?topic=25686.msg154347#msg154347)That post is still valid for me, since I was able to reproduce it with the newest versions available. Only B.2 and A.1 are correct results, B.1 is data loss in the worst case and A.2 creates annoying additional render elements as Corona CXR always does ;)

Good Luck

Hey Frood,

Just to confirm by A.2 and B.1 you mean:

2) BB server local setting autoadd off: 1 layered EXR written (beauty + RE) plus 1 flat EXR (the render element)
and
1) BB server local setting autoadd on: 1 flat EXR (beauty only), NO render element at all

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2020-02-04, 04:47:40
you mean:

Yes, exactly.

Good Luck


Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2020-02-06, 13:39:38
Hi,

Okay so it looks like we can finally reproduce this in one way or another. It seems that there are some issues with saving EXR files.

With the help of Frood, we discovered that this is because 3ds Max saves the EXR saving settings locally on the machine and not in the scene.

This can lead to settings getting mixed up when rendering over a network with backburner. I have sent this over to Autodesk and I'm awaiting their response.

In the mean time, it might be best to set a pre-render script in the scene which sets the EXR file saving settings before each render commences. I hope this can be a suitable workaround for the time being.

I hope this information is useful.

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: zemmu on 2020-03-02, 18:33:58
Hi guys,
I have a similar problem.
Massive exr files BUT all the passes are unreadable.
Is there a way to fix the exr files so I can read them?
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2020-03-03, 17:03:17
Looks like another issue, I would start a new thread for this in "I need help" with a zipped example exr as attachment.

Good Luck


Title: Re: Corona saving layered EXR on render end
Post by: TomG on 2020-03-03, 18:23:53
Yes please on the new thread and example EXR - also though, they are unreadable in what software? (If native Photoshop, it's notoriously bad with EXRs, install the free EXR-IO and use that to open EXRs; if not Photoshop, do mention in the new thread what software it is you are using to open the EXRs).
Title: Re: Corona saving layered EXR on render end
Post by: UsmanS on 2021-02-21, 23:15:31
was there any solution to this issue ?
Title: Re: Corona saving layered EXR on render end
Post by: rowmanns on 2021-02-22, 10:12:34
Hi,

This issue was reported to Autodesk, unfortunately there is nothing we can do about this.

This is a possible workaround:
In the mean time, it might be best to set a pre-render script in the scene which sets the EXR file saving settings before each render commences. I hope this can be a suitable workaround for the time being.

Thanks,

Rowan
Title: Re: Corona saving layered EXR on render end
Post by: UsmanS on 2021-02-22, 23:06:56
Can someone please make this script and post it here?
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2021-02-23, 10:27:25
It is already attached here:

https://forum.corona-renderer.com/index.php?topic=25686.msg158408#msg158408

If you want to enable/ensure to have all render elements in one EXR, you have to uncomment the lines between "-- Switch autoadd on:" and "-- Switch autoadd off:" by removing the leading "--". Then use it as a prerender script.


Good Luck



Title: Re: Corona saving layered EXR on render end
Post by: UsmanS on 2021-02-25, 12:04:52
Thanks for giving me that link, but I couldnt get it to work properly. I basically dont check the box in EXR settings "Automatically add/remove render elements from render dialog", so how do I set up that script so that it doesnt save all the render elements in each file and all files being same size at the end. I want to stay seperate with their original sizes
Title: Re: Corona saving layered EXR on render end
Post by: Frood on 2021-02-25, 12:28:27
I'm not sure if I understand correctly. You do NOT want to have layered EXRs with all the render elements included, but "flat" EXRs for beauty and all render elements - a bunch of 32bit files with 1 layer? You would have to uncomment all lines after "-- Switch autoadd off:" instead. Please compare to the impact/results (https://forum.corona-renderer.com/index.php?topic=25686.msg162908#msg162908). Your case would be 4.B.2 (Corona and EXR -> Autoadd OFF -> Local settings autoadd off). Remember, you only may need this when doing any network rendering like submitting a job to Backburner (Corona DR is not affected).


Good Luck