Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - kizo

Pages: [1] 2
1
[C4D] Bug Reporting / Re: render vanishing after denoise
« on: 2019-02-28, 13:51:50 »
Hi Ben,

just checked and thats it. Very unfortunate and I hope it will be resolved soon.

Thanks on the reply.

P.S. I sent another email to the official support email and no reply agin. Can you please check that?

2
[C4D] Bug Reporting / render vanishing after denoise
« on: 2019-02-28, 10:37:53 »
Hi ,

so I encountered a strange and nasty issue this morning. I have left a render on one machine for a long time. This morning I saved it while it was still rendering as lately I had sveral crashes while denoising.
And fortunately I did so because after the denoise the render is GONE!
VFB is there c4d didnt crash but there is no render shown just black.
If I select a single light select in pass it shows up....so I switch to the LM pass and there is my render back again but its there only for a few seconds!!!
and when saved (while still showing it) it saves a empty black image!!!!

the saved .cxr opens in the image editor and has no issue as the one described above.So I do manage to save the render but its a big issue. saving .cxr files only to save the rendered image is overkill as the file size is huge.

https://www.dropbox.com/s/54nupryoxot6cq5/VID_20190228_102953.mp4?dl=0

this was rendered on a single machine no TR involved. Using Corona 3 hotfix version
It is a very nasty bug and along the numerous crashes on denoising makes work very unpredictable. Add TR not working to these issues and I wonder how this can be considered a working product.

:(

kizo





3
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-12, 10:15:27 »
Hello Moritz,

its great to see you are testing this too. Seems the tests show the same pattern where the faster the render gets the more clients drop out.Makes me think how great and fast would it be if none would disconnect.
Getting no disconnects but a rather slow render time is rather frustrating when you have more power available.

"The main Problem i have with TR right now is, that one has to do a lot of tests to find out the perfect render settings/ number of clients for every project. Almost all time advantage you have with TR is eaten up by finding the best settings for each project"

yes I can agree thats a big problem for me too. It is scene and resolution dependent but also the number of nodes used plays a role so to optimally use the clients one should do extensive testing before rendering and thats taking way too much time

@Tom

I did test a scene from the content browser "Living Room.c4d"  and I got clients disconnecting.I managed to do only one test using all clients for now but will try to do more and let you know.

Cheers

kizo

4
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-08, 17:51:02 »
Im working without TR.In fact I never use it as it always had issue in my tests.Also I have never used it as other engines we use(d) dont require it.
I have Cinema installed on each machine and corona beta.so I just render on a single machine....big stills (6000x4000) overnight or range of animations on each machine at the same time. works excellent.
Now that its a product with license I wanted to avoid buying 10 licences as its a heavy price.


great points jojorender !
I agree a dedicated DR solution would be best choice. Also it would be really great to hear from anyone using TR with success in general even with fewer machines.

5
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-08, 16:58:50 »
yes Tom the no benefit in using the server is base on an actual test...one of the numerous tests we did. the render time is again almost same as on a single 2990wx and the machines rendering have way more power combined

the fewer machines tests were done with 30/100  as I wrote above.

Im glad to hear the devs are thinking about the issues as well as Im very glad on your help. Im only worried how will I manage to work on daily bassis once the betas stop working.
At this point I can see  any improvement will need time. I also see I will not be able to use the nodes that I bought the licences for and I will not be able to use all the costly hardware I have.

very unfortunate and seems I will be forced to look for alternative rendering solution which is a shame as I love Corona very much.

The only other thing I can do is make the 10G network.That is an expensive operation so before investing I would like to know would that resolve the issues for sure?

kizo

6
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-08, 16:33:23 »
TY for the results Kizo! What happens when you don't use all the Clients? (that is, rendering with just 2 clients, just 4 clients, etc., rather than all 10) You may get better results with less clients, as then there won't be network congestion and you won't need to raise the Interval (so you'll keep the speed benefit).

And what happens when using the TR Server approach to have all the machines rendering on that one job?

Hi Tom

rendering wit 2 and 4 clients + main machine is on the above test list too. no benefit at all

7
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-08, 16:32:09 »
Here is another option that may help, which is submitting a job to the Team Render Server rather than using Team Render to Picture Viewer. The downside is that you can't save to CXR, but other than that it should be fine.

The Server may handle sending data back and forth differently, since it is not showing the ongoing progress in the Picture Viewer / VFB, and so may not experience the dropouts - it seems to be independent of packet size, so could be it has its own inbuilt and separate method for handling sending data back and forth between Server and Client.

For me the easiest way to run a job through the Server is
a) Run the TR Server, the Corona License Server, and the TR Clients (including on the master machine, if you want it to contribute to rendering)
b) Use the folder icon in the top left of the TR Server UI to open the local file location where jobs are stored, and then make sure you are in the "admin" folder
c) Copy that address from Windows explorer
d) In C4D, use "Save Project with Assets" and paste the folder location in from above
e) This automatically creates a job for the TR Server
f) Open the browser UI for TR Server using the Globe icon almost top left in the TR Server
g) Start the job. So long as this is only a single frame job, all machines will contribute to it (if it is an animation, each machine will be given a different frame - and tests from Nelaton suggest that this for sure doesn't have issues with either slow rendering or nodes dropping out)

Since I don't have that many machines to test on, I can't say for sure if the possible difference in how the Server handles server-client communication will prevent network traffic problems, but I am kind of optimistic :) If anyone who has a large number of Clients and who experiences these issues can test, I'd be interested to hear the results.

Thanks on the effort but this is a workflow killer even if it worked but it doesnt.
Saving projects locally to be able to render them out each time? thats just crazy
going each time trough the web interface part also.

I mean even if there were no issues its a overly long and frustrating procedure for 1 render let alone doing 20 or more
also there is no benefit in speed using the server.

8
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-08, 16:25:07 »
Hi,

so we have been testing various combinations and suggestions from this thread.

Raising the interval does help with disconnecting nodes. But the overall speed is extremely poor.
These are the specs of the machines used:

      x45   192.168.178.45:5401   PC, 40x2.3GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      x44   192.168.178.44:5401   PC, 40x2.3GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      x43   192.168.178.43:5401   PC, 40x2.3GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      x42   192.168.178.42:5401   PC, 40x2.3GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      x41   192.168.178.41:5401   PC, 40x2.3GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      ws35   192.168.178.35:5401   PC, 16x3.6GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      ws34   192.168.178.34:5401   PC, 32x3.8GHz, 64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle   
      ws32   192.168.178.32:5401   PC, 32x4GHz,    64.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle
         ws31   192.168.178.31:5401   PC, 64x3.8GHz, 128.00 GB RAM, (Studio Client)   Windows 10, 64 Bit, Professional Edition (build 17763)   Idle


We rendered the same 4000x4000px image with different variations

Rendering on the single 2990wx machine takes 14:34

using 8 machines from the list we got these results.The below tests were rendered from the PV with TR

30s/100mb   most nodes disconnected
40/100         1 disconnected        total render time stamped 14:26   (it took 6:20 to gather all the chunks and do the post process)
50/100         0,1 disconnected                                            14:07  (6:01)
60/100         no disconnected                                             15:54  (5:52)
90/100         no disconnected                                             19:57  (7:28)


using 5 machines

30/100         no disconnected                                             14:23

using 3 machines

30/100         no disconnected                                             15:41 


With a packet size higher than 100 times got a lot slower at any interval. When disconnected this is the error we got:

"
Sending chunk 12/13 to the server
Frame synchronization failed: Communication Error"

also this one:


"019/02/08 14:45:42  [Corona4D]
Sending chunk 2/5 to the server
2019/02/08 14:45:42  [Corona4D]
MEMORY_ERROR while sending data
2019/02/08 14:45:42  [Corona4D]
Frame synchronization failed: Memory Error"

with the above error the machine list would still show nodes rendering but the image wasnt updating and at over 20 mins nowhere near done

We also tried starting from the server app using the web interface.This time without the 2990wx. The remaining machines rendered the image in 14:55 The save path was set but the render wasnt save in that location at all.I was saved in the "results" folder.
Th render is in liner color space and extremely different than any of the tests made so far


After this set of tests Im even more worried. It seems that for our network and number of nodes on this particular render the best combination was 50s/100mb as it rendered with 8 machines in 14:07 but compared to the 14:34 of the single 2990wx its very disappointing.

   




9
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-07, 21:54:13 »
Tom I would like to get an answer on the color difference issues also if possible? here is the issue again for easier following but please check the attachments in the 1st post in this thread.

Thanks


"PICTURE VIEWER OUTPUT DIFFERENT FROM VFB

In all the test made so far using TR trough PV (the only possible way) but single machine too,  there is a difference where the PV saved image is always darker.
The c4d project settings are set to linear workflow and sRGB

I also had a situation where the lights didnt match in the render saved as .jpg from the PV to the .jpg saved from the VFB. It seems VFB is showing and saving the lightmix while the PV the beauty pass.That would explain why the color and intensity of light was different.Please check the attached images.
 If the above is true how would one save a non layered format out of PV and have it match the VFB ?
"

Hi kizo,

after reading your description, it seems to me that you somehow managed to save only the non-lightmix version of the image out of the PV. The functionality of the "Save as..." option of the C4D Picture Viewer depends on the current layer mode (Image vs. Single-Pass vs. Multi-Pass) and possibly on what layer you have selected.

As for the different lightness, we are aware of a very slight (almost imperceptible) difference between PV and VFB and it's probably a result of different sRGB handling. But it seems to me (based on your description) that you have a much bigger difference between those two. Might I ask, whether you have any PostProcessing filters enabled? And if so, are the results from PV and VFB the same after disabling PostProcessing?

Hi Houska,

thanks on the help. Im aware of the saving process from PV. I was just saving a file without previously going into single layer mode and choosing a layer.

I made further tests regarding the color and light difference and can reproduce the issue.

Rendering on a local machine only. The scene uses LightMix. Tried both .jpg and .tif.
NO multi pass file set to save so both formats save a single layer file.

1st time rendered in PV and I saved the files automatically from the render settings.
2nd time I rendered from VFB but leaving the save path to save automatically.
3rd I turned off auto saving the render, fired it from VFB and saved manually.

and this is where I get the difference.
the auto saved render from VFB and the manually saved one look different.check the attached .jpgs. tifs look the same

So this is not TR related but I encountered it while testing TR as I usually never render to PV on a single machine.
I have sent the scene I used for testing to Corona support 3 days ago so you can try reproducing it. Please let me know if you manage to test it out.

Thanks

kizo


The render saved from the PV or VFB(set to save in render settings) saves the beauty pass while if saved manually from the VFB it saves the lightmix. At least thats what we concluded after comparing the saved renders and separate passes.It would need checking.

10
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-07, 15:34:22 »
Tom I would like to get an answer on the color difference issues also if possible? here is the issue again for easier following but please check the attachments in the 1st post in this thread.

Thanks


"PICTURE VIEWER OUTPUT DIFFERENT FROM VFB

In all the test made so far using TR trough PV (the only possible way) but single machine too,  there is a difference where the PV saved image is always darker.
The c4d project settings are set to linear workflow and sRGB

I also had a situation where the lights didnt match in the render saved as .jpg from the PV to the .jpg saved from the VFB. It seems VFB is showing and saving the lightmix while the PV the beauty pass.That would explain why the color and intensity of light was different.Please check the attached images.
 If the above is true how would one save a non layered format out of PV and have it match the VFB ?
"

11
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-07, 14:47:33 »
I can't comment on V-Ray and Thearender as I don't know how they were using TR compared to how we use it.

And not necessarily on the packet size, and the smaller ones more often would leave more gaps for keep alives to go in between them. Think of a highway, when lots of small cars are joining the highway, there is always a gap possible by delaying one car to slip in another one (a keep alive), but when a truck made of 18 connected trailers is joining the highway, there is no gap and our keep alive car has to sit and wait for the whole super long truck to get onto the highway before it can join (by which time, it may be too late for it to reach its destination).

(EDIT which is why sending the super large trucks less often may open up gaps for the keep alives to get in there, as one possibility)

a good description of the packet size thing, thanks on that much more suited for my non tech brain.

as for commenting vray or any other engine its a bit political answer. I dont know the ins and outs of the tech behind but I do know I rendered distributed on the same network with 10 and more machines and never had issues. from my POV its a good indication something is wrong with either TR or coronas usage of it.  BTW non of them used TR that might tell something

12
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-07, 14:38:21 »
Hi Tom,

thanks on the further explanation.

Please can you reply to my question:

But if network is the issue how would you explain rendering with vray DR or Thearender distributed a few years back worked without any issues?

also if network is the bottleneck wouldnt it make more sense that it gets congested with far more packets being sent at the same time like when a smaller packet is used?

Thanks
kizo


P.S.  I understand 10 nodes might not be so common and Im willing to test anything you like just to get things working.If you have more specific tests needed just let me know.I have a huge load of work but I will find the time for that

13
[C4D] Bug Reporting / Re: Team render issues
« on: 2019-02-07, 14:14:34 »
Hi Tom,

I will try to make further testing.
But if network is the issue how would you explain rendering with vray DR or Thearender distributed a few years back worked without any issues?

also if network is the bottleneck wouldnt it make more sense that it gets congested with far more packets being sent at the same time like when a smaller packet is used?

in any of these scenarios seems like all my node licences wont be of any use and if the issue is Team render related  I can see the Corona team is pointing to Maxon to solve the issue with TR.
If you research you can see TR issues with other engines as well as C4d ones.So those issues are here for a long time its no news. And that makes the choice to use TR by corona even more strange

as I stated in my emails all I need is to be able to continue the production. As it seems now that wont be possible.

Cheers
kizo

14
[C4D] Bug Reporting / Team render issues
« on: 2019-02-07, 13:28:23 »
Hi,

as some other users I also have issues with TR. The issues are clients disconnecting and difference in output plus other limitations like not being able to save crx format.

As beta will expire on the 21st Feb. we are trying to prepare the workflow but the TR issues just seem to prevent the production.At the current state I feel like the TR is totally useless, unreliable and in fact any node licences are worthless which is a shame for a newly released product. I love Corona and hope to keep using it. We have NO issues working on single machines but distributed rendering or using the full potential of our hardware is essential.
I have been trying to get some help from the official support but unfortunately it takes way too long to get any answer so I will try here.Maybe I will have better luck.



TEAM RENDER ISSUE: CLIENTS DISCONNECTING AND SLOW RENDER TIME

I have spent a lot of time testing things out. Im not a network specialist so its rather hard to understand whats going on and if the network itself is causing issues but I did a series of tests to identify the differences in settings and results obtained.But we used the same machines and network setup with Vray DR and Thea render distributed (CPU+GPU on the network) without similar issues.

We are running a 1Gbp/s network and have a dedicated range of fixed IPs for machines. I have adjusted all firewall setting , disabled bonjour and Im using IP adresses to conect. I have followed the cineversity series of videos on TR and made all adjustments.Corona license server running. Tests we made with the hotfix release.

I made all tests using 9 machines.

From the tests I made so far one conclusion is that the smaller the packet size I set in Corona render settings the less the nodes disconnect.

In fact with a size of 5 or 10 and interval always at 10s none of the nodes disconnected.With a size of 50 they start disconnecting and with 100,200 or even 500 they also disconnect and it seems sooner. The error that we get when the node disconnects is:  "Frame synchronization failed: Communication error" but if pinged or tested the connection the node is available so I can only assume that there were to many chinks at the same time, but Im not sure. Also if thats the case...having smaller chunks would mean a lot more of them at the same time .....but  when chunks are smaller no nodes disconnect at all.

A small chunk size seems to be OK for smaller resolution renders.The tests I made with a 1k square render and small packet size 10 and 10s interval  rendered without disconnecting nodes at all.The scene with all nodes would render to the noise level set in 5-6 min. (same scene on a single 2990wx took 19 mins)

On a 4k square render of the same scene some interesting facts are shown:

A) packet size =  10 mb // render time (stamped)=0:42:00 // no nodes disconnected

B) packet size= 100 mb// render time (stamped)=0:21:34// 4 nodes disconnected (3 almost at the same time some 3-4 mins after starting the render)

Test A took around 1/4th of the time only to collect all the chunks. So it finished rendering in 30 mins and it took 10 to get all the chunks.

Test B  the render finishes in 21min. Even tho 4 of the machines disconnected this is way faster but the issue is not all of the power is used.

I have tried many combinations changing the interval only, the packet size only or combined. Its a very time consuming process. In all my test  nodes disconnect with larger packages wile they dont with smaller ones but than rendering takes way too long.


PICTURE VIEWER OUTPUT DIFFERENT FROM VFB

In all the test made so far using TR trough PV (the only possible way) but single machine too,  there is a difference where the PV saved image is always darker.
The c4d project settings are set to linear workflow and sRGB

I also had a situation where the lights didnt match in the render saved as .jpg from the PV to the .jpg saved from the VFB. It seems VFB is showing and saving the lightmix while the PV the beauty pass.That would explain why the color and intensity of light was different.Please check the attached images.
 If the above is true how would one save a non layered format out of PV and have it match the VFB ?

RENDER TIME INCONSISTENCY

While testing I noticed different render times are reported for the same job.Im not sure why is that but makes it harder to know whats the real render time and estimate.Check attached example.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

It would be great to get any help from the support but Im looking forward to user input too. Anyone out there using TR without the above issues?

Must say Im rather disappointed with how network rendering works. I have 3 x Corona + 3 nodes licences  and one Corona + 10 nodes license along with 10 machines waiting to render.And Im now facing major slowdown in production and cant find a solution. Im not getting any help from the support too.
It feels like the beta stage should have lasted longer even if it was rather long.But making distributed rendering work should be part of a released product and unfortunately cinema Team Render doesnt seem to be the best choice for that.

Cheers
kizo









15
Gallery / Re: Arscom Studio - kizo - gallery
« on: 2017-06-28, 12:17:25 »
Thanks guys! glad you like it
the coastline geometry is all one piece with one 16k texture but I used 2 versions...one for the birdeye shot and a more detailed but smaller one for the "from the sea" view

cheers
kizo

Pages: [1] 2