Ok, sorry for calling it a memory leak - I was too hasty with that definition.
But as I said something definitely wrong here.
I was trying to see better how the ram is growing so limited render threads to 1 first to slow down things. After that I noticed that each thread use 3-4 Gb RAM. With one it's something like 11-13 Gb, with 2 it can go up to 20Gb and then drop to 15-16 and with 8 it's 32-34Gb. So definitely not possible to render scene using 36 threads.
Then I decided to check how obviously problematic object will reflect itself and arranged it in a box like structure, without top (5 instanced objects). Glass facing inwards. Put light and camera inside. Now the RAM usage with just 4 threads is 70Gb +/-.
The first and only candidate for problematic object is the one closer to camera - mirror. Due to the way it's modeled. It's a box. "Glass" assigned to 5 faces and backface is mirror + chamfer.
Same problem if two object intersecting, like "glass" box and "mirror" inside of it.
And then a bit of material investigation - the problem is in glass reflection glossiness (refraction glossiness have no impact). It was set to 0.99. The problem is in range 0.95-0.99+
The higher glossiness value above 0.95 and below 1 - the higher RAM usage and the rendering slowdown, huge.
In recent test I was able to reach almost 100Gb with single thread by using 7 such magical mirrors.
With values like 1 or 0.94 everything working fine with ray depth 100 (difference in RAM between 25 and 100 is negligible, something like 30-70Mb) and the "mirror" IOR 100 or even 999.
So here is your bug as I see it:
highly relfective material (doesn't matter if it's legacy with high IOR or physical set to metal) intersecting or inside of reflective and refractive legacymtl that have reflection glosiness in range 0.95-0.99+ will cause extreme RAM usage and huge rendering slowdown when max ray depth is higher than 50.
And additional problem - you may not be able to cancel it. After some point clicking on cancel doesn't stop the render and max is not responding while memory usage keep growing.
I clicked cancel when usage was 40Gb and after that it was slowly, for probably 30seconds, as I rendered using 1 thread only, growing up to 100Gb and then max crashed.