Author Topic: Deadline task time outs.  (Read 4520 times)

2015-12-18, 11:09:08

j_man

  • Active Users
  • **
  • Posts: 50
    • View Profile
    • Minmud
Hello all,

We're rendering over Deadline and Corona tasks sometimes timeout. By default deadline kills tasks that don't talk for over 8000 seconds (133 minutes) however Corona seems to render a pass every six or seven minutes and the task runs for four hours quite happily before being stopped. Simple fix is to increase the task time-out but what is causing this and what is the correct way to fix it?

Thanks,

J.

2015-12-18, 11:22:26
Reply #1

maru

  • Corona Team
  • Active Users
  • ****
  • Posts: 12711
  • Marcin
    • View Profile
Maybe there is something interesting in the logs? Could you:


    On any of the faulty nodes: go to DrData folder (it can be found in the directory where DrServer is installed on the node)
    Remove the whole content of DrData
    Run the unsuccessful rendering again
    Go to DrData folder again on that node
    Gather the whole content of DrData (you can just pack the DrData folder)
    Send it over

« Last Edit: 2015-12-18, 12:50:49 by maru »
Marcin Miodek | chaos-corona.com
3D Support Team Lead - Corona | contact us

2015-12-18, 12:24:31
Reply #2

j_man

  • Active Users
  • **
  • Posts: 50
    • View Profile
    • Minmud
We're not using DR this time though?

J.

2015-12-18, 12:49:10
Reply #3

racoonart

  • Active Users
  • **
  • Posts: 1446
    • View Profile
    • racoon-artworks
Which version of Deadline are you using?
Corona is officially supported by Deadline now so it should properly update it's status and thus prevent a task timeout to happen. Normally, the render engine communicates with deadline and sends messages to the repository about it's current progress - if this doesn't happen, like with FumeFx simulations (where you have to write a script yourself to do the updating) , Deadline will hit the time out at some point and re-queue the task.
I haven't tested it myself recently and I don't know if it's implemented this way but it should be. You may also have a look at the Thinkbox deadline forum.
Any sufficiently advanced bug is indistinguishable from a feature.

2015-12-18, 12:51:17
Reply #4

maru

  • Corona Team
  • Active Users
  • ****
  • Posts: 12711
  • Marcin
    • View Profile
Oh, sorry, I thought it is Corona's DR managed by Deadline. Sorry, I am not familiar with the software.
Marcin Miodek | chaos-corona.com
3D Support Team Lead - Corona | contact us

2015-12-18, 14:06:45
Reply #5

j_man

  • Active Users
  • **
  • Posts: 50
    • View Profile
    • Minmud
Version 7.0.2.3 R. We will upgrade at some point and try again. Mike from Thinkbox has suggested disabling this for now.

Many thanks for your replies.

J.

2015-12-18, 15:38:14
Reply #6

Ondra

  • Administrator
  • Active Users
  • *****
  • Posts: 9048
  • Turning coffee to features since 2009
    • View Profile
I'd like to quote maru's signature: which corona version are you using?
Rendering is magic.How to get minidumps for crashed/frozen 3ds Max | Sorry for short replies, brief responses = more time to develop Corona ;)

2015-12-18, 16:44:55
Reply #7

j_man

  • Active Users
  • **
  • Posts: 50
    • View Profile
    • Minmud
Hi Ondra,

We're using 1.3, the one with the smiley face < ;



J.

2016-01-11, 11:29:38
Reply #8

Ondra

  • Administrator
  • Active Users
  • *****
  • Posts: 9048
  • Turning coffee to features since 2009
    • View Profile
I had a talk with guys from deadline, and it seems the problem was in too frequent logging in corona. I fixed it, and I would be glad if you would test it in the next daily build
Rendering is magic.How to get minidumps for crashed/frozen 3ds Max | Sorry for short replies, brief responses = more time to develop Corona ;)

2016-01-13, 12:05:10
Reply #9

j_man

  • Active Users
  • **
  • Posts: 50
    • View Profile
    • Minmud
Hi Ondra,

Thanks for looking into it. I have upgraded Deadline to 7.2, which Mike from Thinkbox tells me has a lot of improvements for Corona. I have also disabled the Progress Update Timeout in Deadline and this has fixed the issue.
If I can spare a bit of time I will test the daily build for you.


Many thanks,


Josh.