Chaos Corona Forum

Chaos Corona for Cinema 4D => [C4D] General Discussion => Topic started by: jojorender on 2018-11-07, 21:01:39

Title: Your experience with teamrender
Post by: jojorender on 2018-11-07, 21:01:39

Hi,
I’m currently using Corona version: B2 on mac R18 on server and client.
I’m testing a scene that takes 39:28 min to render to noise level 5% on my main machine.
When adding 1 node in teamrender the render time goes to 29:29 min.
I didn’t expect render times to be cut in half, but definitely better than what I get.
Corona render settings “TR” is set to automatic. Network: wired 1 gigabit LAN.

I plan to use 1 corona license with 3 nodes (if that’s what we’ll get when released)
and send larger jobs that require faster turnaround / more compute power to a render farm. I only do still images.
If the LAN is a bottleneck, would upgrading to a 10 gigabit make sense?
I’m not even sure the gigabit is saturated as it is.

What’s your experience regarding TR?
Any tips on how to minimize overhead / get faster renderings?
How well do additional nodes scale?

Thanks.

Title: Re: Your experience with teamrender
Post by: TomG on 2018-11-07, 21:21:47

Are the machines the same processor, memory, hard drive etc? Or maybe just give the results from each machine using the Benchmark, as there was no info given on their relative performance.

One thing that makes a difference is the size of the packets sent at once - the larger the image you are rendering, the more it may benefit from a different packet size (rather than using automatic). What resolution are you rendering to?

Cheers!

Title: Re: Your experience with teamrender
Post by: jojorender on 2018-11-07, 22:08:43

Thanks Tom,
same CPU, server 64GB, node 32GB ram. I monitored ram usage on node os level - only about 58MB (edit:) swap used.
Both machine benchmark around 4 min.

Test image size: 3000x2000
Tex folder 850MB. Does it make sense to copy project to "Local Repository Path"?
I usually render jobs to 4050x2700px, higher than that and I'm often running out of memory.
What would be better TR config settings for this image size?

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-11-22, 13:46:19

Hi!

Here are my experiences with teamrender so far.

I'm using the latest Build "Beta 2 Final", Cinema 4D R18.057 Studio on a Win10 PC (i9 7940X / 64GB RAM).
My still images are mostly 9000 x 6000. I have 10 Clients at my displosal (8 x i7 7700, 1 x Mac Pro late 2013 with 32GB RAM and 1 x Mac Pro mid 2012 with 16GB RAM) and it seems that these 11 Computers together are about as fast (Maybe half an hour faster) as if'd render it only on my main machine (roughly 3 hours at a 100 pass limit). In my opinion it should at least be half as long if not faster than with one machine only.

Corona Teamrender settings are: Manual/update interval 5s/packet size 10MB. I'm working on my Company's Gigabit-Network.
Looks like sending back all the junks seems to be the most time consuming part but setting them too high caused crashes in the past. Also i experienced a slight decrease in quality when teamrendering compared to single-pc rendering at the same settings/number of passes.

Greetings,
Moritz

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-11-23, 14:03:40

Ok, after rendering a new Project with 20s update interval and 200MB Packet size, it renders way faster.
There is however a quality decrease. 50 passes on teamrender where noisier than 12 passes rendered on my main machine only.

Title: Re: Your experience with teamrender
Post by: TomG on 2018-11-23, 14:34:11

Quote from: HFPatzi on 2018-11-23, 14:03:40

Ok, after rendering a new Project with 20s update interval and 200MB Packet size, it renders way faster.
There is however a quality decrease. 50 passes on teamrender where noisier than 12 passes rendered on my main machine only.

Should be no quality difference - if there is, I'd start a bug report thread, with as much info as possible (including the scene, details about the machines, the OS, version of C4D, version of Corona, the images you are getting). Just to be sure, you allowed time for all data to be gathered from the slaves at the end, and for denoising to run after that? It might take additional time for that to happen (so the main machine may stop rendering, but still be waiting on data before denoising).

Title: Re: Your experience with teamrender
Post by: houska on 2018-11-23, 14:35:28

Hi HFPatzi!

Thanks for your reports! As for the quality decrease though, this should not happen. There were some fixes related to the quality of the rendering in TR, but those should already be included in Beta 2 Final. May I ask you for some pictures, so that we can see the quality decrease ourselves?

Thank you!

Title: Re: Your experience with teamrender
Post by: lenogre on 2018-11-23, 15:43:08

With very huge projects (10 millions of polygons), I prefer make a copy on slave PC because preparation is infinite with TR.
But for small projects, TR works perfectly.

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-11-23, 16:06:05

I will render out two images after the weekend for you to compare. Unfortunatly I can't send you the scene file because of a NDA but i will give you as much information and screenshots as i can get. It is not a big decrease of quality though. But I thought the result should be nearly the same but it is slightly better on a single machine with less passes compared to more passes on 11 machines ;)
I waited until the progress bar in cinema's picture viewer was gone when teamrendering.

BTW is there a way to teamrender a still image in Corona's VFB, since i like that one more?

Greetings and a nice weekend,
Moritz

Title: Re: Your experience with teamrender
Post by: jojorender on 2018-11-24, 00:20:03

Quote from: HFPatzi on 2018-11-23, 14:03:40

Ok, after rendering a new Project with 20s update interval and 200MB Packet size, it renders way faster.

Hi HFPatzi,
thanks for your feedback.
Can you tell us what you mean by “renders way faster”?
Is it now more like you expected? 10 clients a lot faster than the single i9 7940X
I had to look up speed and core count of the CPU’s you use.
i9 7940X  - 14 cores @ 4.3 GHz
i7 7700 - 4 cores @ 4.2 GHz x 8 = 32 cores + the Mac Pro’s (not sure about the specs)
I would also assume that (inc. overhead, LAN, etc) it should definitely be “a lot” faster than the single 14 core.

Questions for the dev team:
Do different CPU calculate pixels differently and maybe contribute to “noisier’ images?
How about a mix of PC’s and Mac’s in this regard?
Can you shine a little light on the manual TR settings and give us some “best practice” advise?   

Happy Black Friday everyone!

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-11-24, 00:47:58

Quote from: jojorender on 2018-11-24, 00:20:03

Quote from: HFPatzi on 2018-11-23, 14:03:40

Ok, after rendering a new Project with 20s update interval and 200MB Packet size, it renders way faster.

[...]+ the Mac Pro’s (not sure about the specs)[...]

Hi,

the newer Mac Pro is some intel xeon 6-core @ 3,5GHz and the older Mac Pro has two 6-core processors @ 2,4GHz.

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-11-27, 12:18:14

Quote from: houska on 2018-11-23, 14:35:28

Hi HFPatzi!

Thanks for your reports! As for the quality decrease though, this should not happen. There were some fixes related to the quality of the rendering in TR, but those should already be included in Beta 2 Final. May I ask you for some pictures, so that we can see the quality decrease ourselves?

Thank you!

Hi houska,

here are the results of my test. I rendered full GI/Path tracing with a pass-limit of 15 and full denoising with the standard settings. Teamrender settings where: refresh rate: 20s, packet-size: 25MB. The upper half is rendered with my i9-workstation only, the lower half is teamrendered with all my renderclients mentioned in another post and the i9 as master. The quality decrease is more visible in areas with bump mapping. You also can see the specs of my render-clients in the screenshot.

I also attached a rar-file with the scene and the rendered images. Hope that helps.

Thanks for your support!

Greetings,
Moritz

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-12-04, 17:30:50

Hi!

Any news on that issue so far?

Greetings,
Moritz

Title: Re: Your experience with teamrender
Post by: houska on 2018-12-04, 17:54:17

Thank you for the pictures and for the scene, Moritz! Your picture really seems like something weird is going on there. We will check your scene out soon and find out what's causing the trouble.

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-12-04, 19:06:03

Thanks for the heads up ;)

Title: Re: Your experience with teamrender
Post by: lollolo on 2018-12-13, 11:33:12

I just rendered a scene two times. I only chanced the settings in the Team Render tab of the render settings.
First:
Client update interval: 10s
Maximum size: 25MB
Time: 26:20

Second:
Automatic
Time: 53:04

The different is just HUGE!
Something is completely strange with the Automatic mode turned on.

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-12-13, 14:59:25

Yeah, I usually bump up the settings to 20s and 150MB. But i don't get the system behind corona's teamrenderer anyways. If you look at the console output it all looks mixed up :D

But I noticed another thing, which might be a reason why sometimes my renderclients just quit (not the whole computer, only C4D-Teamrenderclient) without any errormessage.
This only happens when i render image in final resolution, which is 9000 x 6000 pixel for me. Other (smaller) resolution are rendering without any problems. I also noticed, that there are some communication errors in the console output:

Code: [Select]

2018/12/13 08:23:53  [Corona4D] [TR] Sending chunk 2/5 to the server
2018/12/13 08:23:59  [Corona4D] [TR] Finished chunks: 1/5
2018/12/13 08:24:00  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:16  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:18  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:18  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:20  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:20  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:21  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:22  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:23  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:24  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:25  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:26  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:27  [Corona4D] [TR] Sending chunk 3/5 to the server
2018/12/13 08:24:27  [Corona4D] [TR] Frame synchronization failed: Communication Error
2018/12/13 08:24:29  Peer-to-Peer-Statistik:
    > CAD660023 Downloadgeschwindigkeit 4.83 MiB\s (1x)
    > CAD660027 Downloadgeschwindigkeit 5.75 MiB\s (2x)
    > CAD660022 Downloadgeschwindigkeit 7.45 MiB\s (7x)
    > mac_5s Mac Pro Downloadgeschwindigkeit 10.21 MiB\s (3x)
    > CAD660025 Downloadgeschwindigkeit 10.26 MiB\s (1x)
    > PC01 Downloadgeschwindigkeit 21.11 MiB\s (5x)
    > repronorms Mac Pro Downloadgeschwindigkeit 52.88 MiB\s (4x)
    > CAD660029 Downloadgeschwindigkeit 56.27 MiB\s (2x)
    > CAD660024 Downloadgeschwindigkeit 65.67 MiB\s (6x)
Peer-to-Peer-Statistik Ende
Polycount or Number of Objects do not seem to be a problem here, since my renderclients also quit on a fairly simple scene (1 cube and some deformed and extruded text).
Maybe some of you can proof my thesis.

Greetings,
Moritz

Title: Re: Your experience with teamrender
Post by: CBAS VISUAL on 2018-12-13, 15:49:38

I've the same probleme "[Corona4D]

Frame synchronization failed: Communication Error" when i render image in high def. The clients quit the render without reason.

Title: Re: Your experience with teamrender
Post by: gigabright4d on 2018-12-13, 16:26:44

Quote from: CBAS VISUAL on 2018-12-13, 15:49:38

I've the same probleme "[Corona4D]

Frame synchronization failed: Communication Error" when i render image in high def. The clients quit the render without reason.

I had the same problem too when render the final image (3325 x 4200 px)
the setting was 20s and the maximum size: 150 MB

but when I changed the maximum size to 50 MB, I had no error at all.
So I assume the maximum size is too large that cause the problem.

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-12-13, 16:35:15

Quote from: gigabright4d on 2018-12-13, 16:26:44

So I assume the maximum size is too large that cause the problem.

That's my second guess. Problem with smaller chunks however is the increase in render time.  Need against misery i'd say ;-)

One question though: Has anybody tried the latest corona build?

Title: Re: Your experience with teamrender
Post by: lollolo on 2018-12-13, 16:44:53

Quote from: HFPatzi on 2018-12-13, 16:35:15

Quote from: gigabright4d on 2018-12-13, 16:26:44

So I assume the maximum size is too large that cause the problem.

That's my second guess. Problem with smaller chunks however is the increase in render time.  Need against misery i'd say ;-)

One question though: Has anybody tried the latest corona build?

I guess too. I had the same problem, thats why I changed the maximum size to 25MB.
I'm using the Final Beta. (I think this is the latest build)

Title: Re: Your experience with teamrender
Post by: gigabright4d on 2018-12-13, 17:01:04

Quote from: HFPatzi on 2018-12-13, 16:35:15

Quote from: gigabright4d on 2018-12-13, 16:26:44

So I assume the maximum size is too large that cause the problem.

That's my second guess. Problem with smaller chunks however is the increase in render time.  Need against misery i'd say ;-)

One question though: Has anybody tried the latest corona build?

I agree with you about the render time, larger maximum size renders faster than smaller size.
Ps. I use the latest dally build (corona-3-c4d-daily-2018-12-06)

Title: Re: Your experience with teamrender
Post by: Beanzvision on 2018-12-14, 07:44:26

Quote from: loliolg on 2018-12-13, 16:44:53

Quote from: HFPatzi on 2018-12-13, 16:35:15

Quote from: gigabright4d on 2018-12-13, 16:26:44

So I assume the maximum size is too large that cause the problem.

That's my second guess. Problem with smaller chunks however is the increase in render time.  Need against misery i'd say ;-)

One question though: Has anybody tried the latest corona build?

I guess too. I had the same problem, thats why I changed the maximum size to 25MB.
I'm using the Final Beta. (I think this is the latest build)

The latest build can be found here guys:
https://drive.google.com/drive/folders/17msYVIuJPs___3w192dpuyeF2MjW1B__

Title: Re: Your experience with teamrender
Post by: HFPatzi on 2018-12-14, 11:35:46

I just tested teamrtendering (9000 x 6000 px) with a max size of 50MB and a refresh-time of 20s and it renders without any problem. So for a start, i'll teamrender with 50MB max size. But i have to say, the whole corona/teamrendering topic is a little bit confusing ;)
A easier to use solution in the final version would be much appreciated :)

Have a nice weekend all together!


________
Edit:

While finishing the teamrender (waiting for chunks) i noticed that some of the clients finished way earlier than others (same hardware). As i checked the console-outputs of them, it seems that some network communication errors happend. Don't know if this is something i should be afraid of but anyhow attached 2 example logs. One from a client with and one from a client without network errors. I just censored some of the asset names ;)

Title: Re: Your experience with teamrender
Post by: jojorender on 2018-12-16, 22:02:55

Did you guys notice that these client disconnects happen right around the same time the "low memory" warning pops up?

I tried all settings 20s/150MB - 10s/50MB on a image 4500x3000 using only 1 client machine to rule out a network bottleneck.
Every setting i tried had disconnects. Not easy to test, since it always takes a while till it happens.