Rendering 360 stereo for VR live action compositing

[wilbertvandenbroek]

Hey guys I'm new here to this forum and a trial user of Corona. I've been testing Corona and thus far I'm pretty impressed by its speed, possibilities and ease of use.

Here at Isaac VR we have a new project were we need to integrate 360 stereo CG into a shot live action plate. The live action is filmed with a Nokia OZO.
I'm having great trouble in matching the placement of a cg object into the plate. Here is my render setup.

1) I used a single Tiff taken from the live action footage which has a Top (Left Cam) Bottom (Right Cam) configuration at a 3840x2160 resolution.
2) I used this Tiff as and (SDR) environment input set to spherical
3) With a Physical cam with the CoronaCamMod I used the spherical projection and stereo override.
4) Now any object that is in the scene will be rendered Top bottom and with stereo separation.

So far so good you might think but here comes the weird part. The Top image that is being rendered is the same as the bottom. It looks like the bottom half of the input TB tiff is being used  twice. It looks like the bottom half is reprojected to the top with stereo separation. Needless to say combining the CG with the plate in post doesn't work. The bottom half of the image lines up correct with the plate, the top doesn't match. In my project case an object floats in mid air instead of standing on a table.

Does anyone here have an idea on how to solve this. Corona does something behind the scenes that I can't control.
Only thing I can think of to fix this is separating the left en right image and processing them individual after which you combine them in post production.

Thanks in advance,

Wilbert

[johan belmans]

Hi Wilbert

Two questions: are you using Max? What distance did you use for eye separation?

[wilbertvandenbroek]

Hi,

Yes I'm using 3ds max. The Eye separation I kept at default settings 6.3 for testing.

[wilbertvandenbroek]

Here's a step by step workflow image of the problem I've encountered.

[romullus]

Uncheck tiling in bitmap node, that should solve your problem.

[wilbertvandenbroek]

Unfortunately that didn't solve the problem. The issue that I need to fix is that the stereo option in the camera mod duplicates the the complete hdr to the top of the stereo render. So any change made to the complete hdr (e.g. tiling or offsetting) will be copied to the top. Maybe my approach to composting cg into a 360 live action is completely wrong.

If anyone can point me in the right direction that would be great.

[TomG]

I always think of the top / bottom (ie left eye / right eye) images as two separate renders that happen at once - which means any background will indeed appear in full in both (rather than it being one render, where the background would be stretched across both left and right eye results)

Never tried compositing onto an already-stereo image, but my first thought would be to render without the background, just getting the 3D object with an alpha, and recombine in post (stereo output of 3D object with background as an alpha on top of the stereo footage.) A non-stereo version of the background would be used to get reflections etc, and a shadow catcher for catching shadows onto the table.

Have never tested it though - may give it a try and see!

[wilbertvandenbroek]

Hey Tom,

That is exactly what I have been trying. A render with top and bottom in one image doesn't work because I can't match both top and bottom in one go. I'm trying a setup in which I split left and right (e.g. top bottom) from the original plate and process them separate. I'll keep you updated on the progress.

[wilbertvandenbroek]

Tried it with two different scenes and it's still not working. Using left and right image for both scenes and matching it in the render (positioning the object so that it sit correct on the table) I got a huge difference in the 360 viewer. Objects on the table were correct, objects on the piano totally wrong...
I'm running out of options here.

[johan belmans]

Hi Wilbert

I could not react sooner wasn't at the office.
I do not have any experience with combining real life footage and Corona VR.
But maybe the following thread started by Jcdeblock can be of any help.
https://forum.corona-renderer.com/index.php/topic,11061.0.html

Good luck

[wilbertvandenbroek]

Thanks, I'll see if I can contact him he also lives in The Netherlands :-)

[wilbertvandenbroek]

Just a quick update. I fixed the problem by separating the left and right eye and process them apart from each other. Then in comp I join them top bottom and it's done. To correctly add stereo separation you have to add real live trackers to your shot footage or eyeball the sh*t out of it :-), the first being the easier way. Thanks for all your help.