Render to Texture issue (Normal Map Raytacing issue?)

[Cobra8472]

Hi Everyone,

Our texturing process is built upon baking in lighting information and data into a flat diffuse.
I'm running into an issue, where normal maps when doing a diffuse render to texture are providing incorrect surface information.

It may be easiest for me to illustrate via some simple imagery.

Here is a simple viewport render with a simple reflective material and HDRI background:
www.leatherneck-sim.com/RTT_Bug_01.jpg

Here is the exact same setup, rendered to texture. Note the seams on the front.
www.leatherneck-sim.com/RTT_Bug_02.jpg

If I unify the normals on the part in question, I get the same issue in the viewport render. See image below.
www.leatherneck-sim.com/Issue_01.jpg

Here is a more illustrative view of the issue in the RTT window:
www.leatherneck-sim.com/issue_02.jpg

This leads me to believe it may be something smoothing group/vertex normal/face normal related?
Perhaps there is a difference in how the surface normal is used in RTT? It happens on almost every surface, it's just most obvious on the front corners.

This happens with: Corona 1.4, Vray 3.2, MentalRay in 3Ds Max 2017.
Here is a image for reference of the complete asset: www.leatherneck-sim.com/HUD.jpg

Many thanks in advance for any help,

Nick

More info on the issue and scene to reproduce are in #24 post
romullus

[romullus]

Corona doesn't support projection mapping RTT, so you have to bake your normal maps elsewhere. You can read everything you need to know about normal maps here: http://wiki.polycount.com/wiki/Normal_map an here: http://wiki.polycount.com/wiki/Normal_Map_Modeling

[Cobra8472]

Hi Romullus,

Thanks for the reply-- but this is not about baking normal maps. The normal maps is done, baked by another tool and applied to the object.
The issue, in short: The normal map in render to texture is rendered incorrectly on the object. It works fine in the viewport.

[romullus]

Can you tell how normal maps on your model looks when doing regular rendering, not RTT? Maybe you have somerender time modifiers that affects smoothing on your object?

[lacilaci]

This issue looks like exactly what happens when you don't use cage for normal projection or average normals function(substance painter) the curvature of that rounded edge is being projected on the planes from 90 degree angle instead averaging it towards the edge.

Are you sure your normals aren't showing those seams elsewhere, only after RTT?

[maru]

Since the issue is visible near seams, could it be something with padding? Have you tried rendering with different padding settings? ( RTT > Objects to Bake > Selected Object Settings > Padding - I would check with 0 and values higher than the default 2)

This happens with: Corona 1.4, Vray 3.2, MentalRay in 3Ds Max 2017.

So it means that this is not something Corona-specific, right?

[Ondra]

padding should never influence anything inside the geometry, only fix the filtering seams, My guess would be some problem with normals

[Cobra8472]

Sorry to bump such an old topic; but this is still an issue-- I was just completely swamped to follow up on this topic.

It will happen to pretty much any object with baked down normalmaps (i.e. HP -> LP normalmaps)

I have tried almost every combination of seam setup in order to figure out the issue; but it simply seems to be a limitation in the way RTT handles surface normals?
Any advice would be greatly appreciated!

[romullus]

AFAIK RTT in 3ds max doesn't have any issues with baking normal maps. It may be not the most convenient baking tool to use, but it does its job well. Of course there are some rules how to properly bake normal maps and those rules should be respected in every baker. If you're not familiar with them, i strongly recomend to stop any baking attempts untill you'll learn proper techniques. I'd suggest you to start from here: http://polycount.com/discussion/107196/youre-making-me-hard-making-sense-of-hard-edges-uvs-normal-maps-and-vertex-counts and continue your research further, either on polycount or anywhere else on the web.

[Cobra8472]

Thanks for the reply! but again;  this is not about normal map baking techniques. My normal maps are perfectly correct, and render correct in custom engines, UE4, Marmoset, Unity, et al- I'm extremely well versed in normal map baking, and I've done bakes in everything from zbrush to substance and the new marmoset 3.

This seems to be an issue with something like the camera projection matrix or surface normal directions when an object is flattened by the renderer to RTT.
It's the best way I can explain it. Oddly enough; the issue is non-Corona specific (meaning several renderers have the same issue), alas I hope that there is a workaround.

Take a normal mapped (from HP, smooth edges) Low Poly cube: put a reflective material on it, do a RTT of the object, and the resulting object will have seams (whereas rendering the object in perspective or camera view, it appears perfectly normal)

[Cobra8472]

Ah, crap, I just realized I had broken the image links.

I will make a new (much simpler) example ASAP.

[maru]

Hi, just reviving this.
Cobra8472, can you provide some super simple example images demonstrating this, with a step-by-step guide?
I know that there is already a description here, but images always help greatly.

OTOH if this is happening in other engines too, my first bet would be that it's a Max issue...

[Cobra8472]

Hi Maru,

Sorry for the delay-- development crunch ramped up and I worked around this by using the Max scanline renderer.
I am now running into this issue again.

To be very clear, here is a sketchfab of the mesh, illustrating that the normals appear perfectly fine in a realtime renderer. They also appear perfectly fine in a normal render in Corona.
https://skfb.ly/6sNET

Seams however appear on mesh seams when doing a Render-To-Texture, even when it otherwise appears perfectly in the Corona renderer.
See this image: http://i.imgur.com/4a7A2Y9.png

I've uploaded a package with this model, normalmap and Max 2017 scene here: media.heatblur.se/RTT_Issue.rar


Appreciate any help!

All the best,
Nick

[romullus]

I still don't get it. If you already have normally working normal map (judging by sketchfab viewer), then why you're trying to rebake it with Corona RTT? I can't look at your scene, since i'm on max2016.

[Cobra8472]

I'm not trying to bake a normal map.

I'm baking a diffuse (Beauty) map. The Normal map does not render correctly  when doing so, even when it is fine in a normal camera projection.

[romullus]

Sorry, i still don't get it. Could you resave that scene in your previous post to max 2016 and reupload it? Maybe then i can figure out what's going on.

[Cobra8472]

http://media.heatblur.se/RTT_Issue_2016.rar

The procedure is simple: Render the scene from any angle and note that the object appears just fine (like in Sketchfab).

Run the render to texture which is setup, save the texture, and apply it to the model.  Note all the seams appearing due to incorrectly shaded lighting and reflections.

To the corona team: this may be something with how normals, binormals or tangents are calculated in RTT mode.

[romullus]

Server not found...

[Cobra8472]

Fixed!

[romullus]

"The file was saved in max2017"

Thanks for wasting my time...

[TomG]

I downloaded the file, saved it as Max 2016 from 2017, here it is attached -

[romullus]

Thanks Tom, did you save it with 1.7? - it crashes max on trying to open with 1.6, but opens fine with daily build installed. Anyway, there was no normal map texture with file, so i couldn't test what's wrong with RTT, but the model itself has big problems - first of all it has smoothing groups scattered all over the place and that's more than likely will cause problems with normal maps. What's more starnge, is that Corona RTT throws error message that model contains no triangles and it will not render. I tried all the tricks that i know to fix that, but the only one that helped was exporting model as OBJ and reimporting it back.

My conclusion at the moment is this: Corona RTT works fine, problem is in the model.

[TomG]

Oops sorry my bad, yes I had the latest daily installed :(

[Cobra8472]

Thanks Tom, did you save it with 1.7? - it crashes max on trying to open with 1.6, but opens fine with daily build installed. Anyway, there was no normal map texture with file, so i couldn't test what's wrong with RTT, but the model itself has big problems - first of all it has smoothing groups scattered all over the place and that's more than likely will cause problems with normal maps. What's more starnge, is that Corona RTT throws error message that model contains no triangles and it will not render. I tried all the tricks that i know to fix that, but the only one that helped was exporting model as OBJ and reimporting it back.

My conclusion at the moment is this: Corona RTT works fine, problem is in the model.

This is getting redicilous. This bug is super obvious and we're now two pages in and you are still telling me that my model / normal is wrong. I appreciate the effort to help, but you need to stop telling me that I am so incompetent that I cannot create a properly baked normal mapped object.

It's disappointing that Corona staff is not commenting. The issue is readily apparent within 5 minutes of: baking a model with a normal map, and doing a RTT (Beauty map). Seams all over the place on the rendered texture.

With regards to your specific concerns:

1) The smoothing groups are scattered because that is what you do for a normal baked model: you split the model by UV Seams. This is industry standard!!
This causes NO problems in any realtime engine: Unreal Engine 4, Unity, 3ds max with normal mapping shader. Not a single engine has any issues (you just commented on the sketchfab above, which has the exact same smoothing groups)

2) The model containing "no triangles" is because Max RTT tends to default to MatID/UV Set 3 for RTT. Set it to 1 (where the UV is) and it works without issue.

3) The Normal map is included in the previous .rar, where the file was in Max 2017 format. It's in DX Tangentspace (IIRC) - but try flipping green just to make sure (I can't recall off the top of my head)[/list]

[Cobra8472]

I've made a completely new example now so we can move away from the model blaming. As simple as you can get: a box.
A Low-Poly box with a HP baked normal. Smoothing groups split along UV seams. Standard.

Here is what it looks like in Marmoset without the Normal applied: http://i.imgur.com/bqBul21.png

Here is what it looks like in Marmoset with the Baked Normal: http://i.imgur.com/F3FJlzf.png
Good.

Here is what it looks like in Corona with a Camera render with the Baked Normal: http://i.imgur.com/xLJpq6o.png
Good.


All is good so far


And now lets run a Render to Texture (Corona Beauty) with appropriate padding.
Apply the rendered texture to the model and switch to Flat viewport mode to remove all lighting:

Here is how it looks: http://i.imgur.com/89FIK7l.png
Bad!

There are now seams at UV borders!! The rendered texture has seams.
Everything is identical, but the RTT texture has seams.



Here are all of the files if you want to try by yourself.
The .max file is for sure in Max 2015 format: http://media.heatblur.se/CORONA_ISSUE.rar

[romullus]

Sorry, didn't realised that you have splitted smoothing groups along uv shell borders. Smoothing is messy, but if it bakes well, then that's no problem. I will take a look at your box example tomorrow.

[romullus]

Ok, i reproduced and can confirm the bug. Moving this topic to bug reporting board. Thank you for scene.

[Cobra8472]

Thanks for the help romullus! Apologies for the frustrated tone. Stressed! :)

[romullus]

No worries, with the proper scene it was a matter of seconds to replicate and confim the bug.

[Cobra8472]

Corona team; any ideas/progress on this issue?