Is there a technique/tool to update render nodes?

[Jpjapers]

Im wondering how people manage render farms. We only have 8 nodes but it would be useful to know if theres a way to batch update their software rather than have to go through each node and run updates and installers. Anyone got any ideas?

[Frood]

Well since we use and test DBs in production, I maintain a deploy script (not even powershell) which reads nodenames from another file and then does for every node basically the following:

- stop render service (I run drserver as windows service)
- wait for max intance(s) to terminate
- clean drdata directory
- clean plugcfg directory
- copy drserver executable if present
- copy corona files
- start render service again

So a new release / DB / RC means just 1 minute actual work for me (extract files, increment directory name, run script), the rest completes in the background in about 3 minutes for ~20 nodes.

There are a few requirements though (most important of course that every max installation has to be exactly the same regarding paths and admin share access with the same credentials on every node must be granted) which prevents me aside other things to share it but I try to make it more and more generic with the purpose of having some day exactly what you are asking for :)

Here is a screenshot of the output (older version):
https://forum.corona-renderer.com/index.php/topic,11657.msg75326.html#msg75326


Good Luck

[Jpjapers]

Well since we use and test DBs in production, I maintain a deploy script (not even powershell) which reads nodenames from another file and then does for every node basically the following:

- stop render service (I run drserver as windows service)
- wait for max intance(s) to terminate
- clean drdata directory
- clean plugcfg directory
- copy drserver executable if present
- copy corona files
- start render service again

So a new release / DB / RC means just 1 minute actual work for me (extract files, increment directory name, run script), the rest completes in the background in about 3 minutes for ~20 nodes.

There are a few requirements though (most important of course that every max installation has to be exactly the same regarding paths and admin share access with the same credentials on every node must be granted) which prevents me aside other things to share it but I try to make it more and more generic with the purpose of having some day exactly what you are asking for :)

Here is a screenshot of the output (older version):
https://forum.corona-renderer.com/index.php/topic,11657.msg75326.html#msg75326


Good Luck

Well damn...i didnt think of just copying the drserver file. Ive been running the installer manually on each one.
Thats a much better idea haha thanks man :D

Would you be able to go into some further detail on your script and or method? im no coder (well a little VB.net and maxscript) but the screenshot you linked to looks like what youve set up is pretty slick.

[Frood]

i didnt think of just copying the drserver file.

I assume (and hope) you are aware that you have to do the rest ("copy corona files") as well to install another version of Corona properly, DrServer.exe alone is just the spawner.


Good Luck

[Jpjapers]

i didnt think of just copying the drserver file.

I assume (and hope) you are aware that you have to do the rest ("copy corona files") as well to install another version of Corona properly, DrServer.exe alone is just the spawner.


Good Luck

Thinking about it properly i feel very stupid right now haha

[jasond]

Wish I saw this thread earlier... I just manually installed Corona 1.6 on 15 render nodes :(

Frood, how are you checking that the drservice and max instances are stopped before copying over the files and starting the service again? We're using nssm but from outside the Corona folder and robocopy from network drive to node drive unc.

Thanks,
JD

[Frood]

Frood, how are you checking that the drservice and max instances are stopped before copying over the files and starting the service again?

Here are the loops, redirection of stderr and stdout is just to avoid spamming of the output and to log into a file and "%*" is the variable iterating through the node names. The rest should be obvious.

Service control (sc.exe) for the service:

Code: [Select]

rem #stop the dr service
sc \\%* stop %servicename% 1>NUL 2>>%errorlog%
goto querystopped

:stopdelay
echo * Waiting for %servicename% to stop...
timeout /t 2 /nobreak >NUL 2>>%errorlog%

:querystopped
sc \\%* query %servicename% | find "STATE" | find "STOPPED" >NUL 2>>%errorlog%
if errorlevel 1 goto stopdelay
echo * Service stopped.
echo.

... and takslist for the process check:

Code: [Select]

rem #waiting for max to terminate
:waitformax
echo * Waiting for Max process to terminate...
timeout /t 5 /nobreak >NUL 2>>%errorlog%

tasklist /s %* | find /I "3dsmax.exe" >NUL 2>>%errorlog%
if %errorlevel% == 0 goto waitformax
echo * Max down.
echo.

But if there has been only drserver running, waiting for 3dsmax to terminate got obsolet in the latest dailies and the final (1.6) because obviously it get´s ungracefully terminated by Pavels sledgehammer ;] But there may be other instances running which would prevent an update so I left it like this.


Good Luck

[Frood]

Would you be able to go into some further detail on your script and or method?

Sorry, seems that I missed your edit. Is there something specific left you want to know? The loop above for drserver works similar to start the service again, cleaning directories is just deleting logfiles, configs, drDumps, history files and max files if present, then copy over the files to %maxinstalldir% and %drserverinstalldir% (both as variables to meet the installation), that´s it basically.

This is the custom section of the script, already prepared for a somewhat generic usage:

Code: [Select]

rem --- ! Customize this ----------

rem Where to find the extracted Corona install files:
set releaseroot=.
rem set releaseroot=.\_dailies\170421

rem Which subdirectory should be used:
set releaseversion=3ds Max 2016 Plugin\fullspeed

rem The directory where max is installed on your nodes:
set maxinstalldir=C$\Program Files\3ds Max 2016

rem The directory where CoronaDrServer is installed on your nodes:
set drserverinstalldir=C$\Program Files\Corona

rem The name of the file with a list of nodes to process, one per line:
set nodelist=nodes_to_update.txt

rem The name of the DrServer windows service:
set servicename=CoronaDrServer

rem The user that runs the windows DrServer service:
set serviceuser=system

rem Error log filename
set errorlog=CoronaUpdateErrorLog.txt

rem If the error log should be opened after processing
set openerrorlog=1

rem If the error log should be initially deleted
set keeperrorlog=0

rem ---- End customization ----------


Good Luck

[jasond]

Frood,

Finally got around to implementing your code, thanks so much for the examples! This saved me quite a bit of time.

One thing I noticed is that even if the service query says DR is stopped, it still keeps it in memory for a bit and got a sharing violation. I had to put a 5 second delay in there before copying files. Could just be from the tight IT security here though.

Thanks again,
JD

[Frood]

Violation during copying? Well yes, depending on Corona build, the max process may need a while to terminate after stopping DrServer service, but if you check for the max process as well, this should not happen.

But glad it's working for you, so no "retarded technology" (Ondra about an msi package to throw on the domain controller for automatic slave/workstation installation) is needed ;)

Next generation Corona installer with the option of a silent default install may change the game again, let's see. Still no DR server-as-windows-service support I presume so I will most likely keep the script solution.


Good Luck