2
« on: 2021-08-28, 12:38:47 »
Hi.
Recently working on a heavy scene, I've noticed that my render nodes are taking quite some time to pick up the job and actually starting to render. Talking about 5-10 minutes on a 2.1 GB scene and 2.24 GB assets across 444 files.
So, everything is pulled out from a pure SSD NAS running over 10Gbe. The render nodes uses "standard" SSD local disks (app. 500/500 R/W performance)
All the assets in Max have the appropriate UNC path set, and all the nodes uses the same account name and can see/read those files.
What I've noticed is that in the VFB the nodes spend quite some time on "downloading assets"?
Can anyone shed some light on how exactly does DR internally work?
1. Prior being sent to the nodes, the working .max file gets saved locally or in the original opened location? I'm asking this because even if the 10Gbe link get's saturated between the workstation and the NAS in normal network traffic, it's still faster to save the scene on a local NVMe drive running on PCIe. and copy that large file to the NAS.
2. What does "PARSING SCENE" mean? I'm guessing that's the time required to actually open the file and load all the assets on the slave machines?
3. "DOWNLOADING ASSETS" - does this refer to actually copying the assets on the target machine, or it's the action where the node reads the assets from a shared network location?
Here's an extract from tone of the render nodes logs:
2021-08-28 11:35:04 Ready for new connection
2021-08-28 11:42:39 Accepted remote connection from 192.168.2.30
2021-08-28 11:42:39 Started connection #3
2021-08-28 11:42:39 Handshake: <root><DataType>0</DataType><Handshake>init</Handshake><FrameName>0</FrameName><HostVersion>Max2021</HostVersion><CameraName>CoronaCamera001</CameraName><Region>0 0 0 0</Region><ImageSize>1800 1350</ImageSize><MD5>47b6082ca67f9a723bd7e7dd7d2bd227</MD5><CoreVersion>7 (Hotfix 1)</CoreVersion><IpAddress>192.168.2.30</IpAddress></root>
2021-08-28 11:42:39 Saving incoming scene to C:/Users/backburner/AppData/Local/CoronaRenderer/DrData/received_scene_1841907443.max
2021-08-28 11:42:49 Downloaded scene: 2.161 M bytes in 4 s
2021-08-28 11:42:56 Received sampling focus mask (region 0 0 1800 1350)
2021-08-28 11:42:56 Render for connection #3 started
2021-08-28 11:45:37 Received a list of missing assets
2021-08-28 11:45:37 Got a message from remote: all_assets_sent
2021-08-28 11:45:38 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:45:38 Received a list of missing assets
2021-08-28 11:45:39 Got a message from remote: all_assets_sent
2021-08-28 11:45:39 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:45:40 Received a list of missing assets
2021-08-28 11:45:40 Got a message from remote: all_assets_sent
2021-08-28 11:45:40 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:45:41 Received a list of missing assets
2021-08-28 11:45:41 Got a message from remote: all_assets_sent
2021-08-28 11:45:42 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:45:42 Received a list of missing assets
2021-08-28 11:45:42 Got a message from remote: all_assets_sent
2021-08-28 11:45:43 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:46:21 Received a list of missing assets
2021-08-28 11:46:22 Got a message from remote: all_assets_sent
2021-08-28 11:46:22 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:47:09 Received a list of missing assets
2021-08-28 11:47:09 Got a message from remote: all_assets_sent
2021-08-28 11:47:10 Sending command to slave: <root><Command>all_assets_sent</Command></root>
2021-08-28 11:50:11 Started EXR dump (402.082 bytes, filename: C:/Users/backburner/AppData/Local/CoronaRenderer/DrData/dump4306.exr)
2021-08-28 11:50:11 Finished EXR dump after 0 s
2021-08-28 11:50:12 Received sampling focus mask (region 0 0 1800 1350)
2021-08-28 11:51:13 Started EXR dump (202 M bytes, filename: C:/Users/backburner/AppData/Local/CoronaRenderer/DrData/dump4307.exr)
2021-08-28 11:51:14 Finished EXR dump after 1 s
2021-08-28 11:51:14 Received sampling focus mask (region 0 0 1800 1350)
2021-08-28 11:52:16 Started EXR dump (204 M bytes, filename: C:/Users/backburner/AppData/Local/CoronaRenderer/DrData/dump4308.exr)
2021-08-28 11:52:16 Finished EXR dump after 0 s
2021-08-28 11:52:16 Received sampling focus mask (region 0 0 1800 1350)
2021-08-28 11:52:39 Got a message from remote: terminate_hard
2021-08-28 11:52:40 Sending command to slave: <root><DataType>0</DataType><Command>terminate_hard</Command></root>
2021-08-28 11:52:41 Slave 3ds Max ended render
2021-08-28 11:52:41 DR is finished
2021-08-28 11:52:42 Ready for new connection
I can see the the scene was copied in 4 seconds which is as expected, but can't understand why it took almost 5 minutes to actually start working on the job?
And why are there 7 entries for "Received a list of missing assets"?
Upon further examination of the DrData folder on one of the remote machines, the only files there corresponding to that job are the .max file , 2 config files, and a couple of small files from Forest Pack dist maps and custom CUBE files from the frame buffer.
Since I've switched from Vray a couple of years ago, I can't do a comparison test, but I recall scenes would load much faster on the render nodes using Vray DR back at the time.