Corona masks RGB no clean edges why?

[Light&Bit]

Hi chaps,

as you can easly understand from the subject of my topic I am getting mad when I have to extract the RGB mask passes from the render elements in Corona, especially if I have trees (and this means opacity channel with a bitmap for the leaves) in my scene I have noticed the edges are not cleaned at all, there is always a border of 1-2 pxl (I have already tried to save them in different formats, tiff, tga, png but always the same problem) and this is quite frustrtating because there are useles if I have to do PP in photoshop.

any suggestion about how fix this issue?

thanks in advance

Dan

[mraw]

I think I know what your problem is. But to be sure I understood right: Are you using a Mask RE for your trees or have you set up an RGB-pass: A render with 'solid' red, green and blue shaders combining 3 masks within an rgba-render?
Screenshot would be great, too. thanks

[Light&Bit]

hey mate,

I have assigned an objectd Id for each object and then added a CMasking_Mask in RGB mode, please see attachement. with Vray I have never had this issues.. :/

[Light&Bit]

missing one attachment sorry

[mraw]

I would recommend to use Mask RE, not the RGB-solution, because you might get in trouble when objects overlap. But from the last screenshot I see that you're not rendering on a black BG?
BG always have to be black, otherwise you will get fringes in the edges.

[Light&Bit]

so this means everytime I have to render out the masks i have to re-render the scene replacing the BG with just a black solid color? mh.....let me try, thanks  for the suggestion.

Best,

Dan

[mraw]

You don't have to rerender. You render your beauty with a clean alpha(on black!) and then you add your sky/bg plate in post.

[Light&Bit]

mh...ok I am not following you anymore.

I don't need to replace the sky, for example I need to tweak the curves/levels for few objects, like the trees.

So even after I tried to render out the masks with a black BG (I have to use the option "Render only masks" because otherwise Max runs out of Ram and crashes ) the problem is still the same the edges of my masks are not clean and sharp and they are not able to work properly in photoshop.

[mraw]

https://en.wikipedia.org/wiki/Alpha_compositing
or do a bit of googeling 'premultiplied alpha'. I know it is a bit of pain in the ass, but beneficial once you get your head around it.

Can you check the bitmaps you are using for the leaves? Maybe the problem with the alpha is there.

[Light&Bit]

I know how fix it easly in AE or Nuke, but in Photoshop is a bit limited. but the point is it must be come out correct for Max and Corona somehow.

[pokoy]

I know how fix it easly in AE or Nuke, but in Photoshop is a bit limited. but the point is it must be come out correct for Max and Corona somehow.

It's actually a limitation of PS. You can use the defringe command in PS but it's destructive. There used to be a site with another way to set it up but it's not online anymore and I don't remember the details. But this also involved deleting image portions.
Another idea is the Mask Edge command which can be really useful and may get you there.

[Frood]

It's actually a limitation of Corona :)

At least additionally. Because unfortunately you don't get a proper mask as soon as opacity is used.

This is not about aliasing or rendering on non-black/premultiplying. What dboldicotti wants to do is impossible right now with simple render elements because all of them: refraction, reflection and yes - even opacity (used here) is not handled correctly by masks at the moment.

Here is some kind of workaround - reminding of dark mr ages where you always had to have some additional "render elements scene":

https://forum.corona-renderer.com/index.php?topic=17085.msg106956#msg106956



Good Luck

[Light&Bit]

Straight to the point, thanks Frood to confirm my  doubt about the masks elements.

Best

Dan

[sebastian___]

I can't be sure how it would work in Corona, and I didn't tried it for the last few years, but in the past I used to export passes with tga and uncheck "Pre-multipled Alpha". In every viewer like XnView the picture would look strange, but in Photoshop and After Effects the edges would look just like it should. In AE I think you need to right click the image and select Interpret Alpha and select Unmultiplied or something.



Another benefit - if you leave "Compress" checked - that's lossless (meaning identical to uncompressed) but very fast compression - so good for playback.
 So Compress ON , Pre-multipled Alpha OFF

[mraw]

@Dan. Sorry man. I just saw your render against a nonblack BG and immediately thought of a premultiplying issue. Users complaining 'can't get rid of this fringe' do usually something wrong with the alpha.

I am not able to do some tests, because I have no active license at the moment. But as I understood Frood: At the moment you can't have a tree with 'sprite' mapped leaves and get a working RE mask for that? Is that right?

I can't be sure how it would work in Corona, and I didn't tried it for the last few years, but in the past I used to export passes with tga and uncheck "Pre-multipled Alpha". In every viewer like XnView the picture would look strange, but in Photoshop and After Effects the edges would look just like it should. In AE I think you need to right click the image and select Interpret Alpha and select Unmultiplied or something.



Another benefit - if you leave "Compress" checked - that's lossless (meaning identical to uncompressed) but very fast compression - so good for playback.
 So Compress ON , Pre-multipled Alpha OFF

It is about to know what you're target application expects: pre or unpremultplied. And you usually can tell your software how to treat the imported footage as well.