So the problematic object is a box, so one object with a MultiMaterial assigned? Not different objects for each material?
In this particular scene - yes, it's a single object, with a MultiMaterial, a kind of real life mirror - backside is reflecive and rest is glass.
If I assign one material, glass or mirror to that object - all good.
But same problem happens with different objects. So, I just cloned that object, assigned glass to 1st, mirror to 2nd, then push modeifier to 2nd, so there is no overlapping faces and moved it back a bit, but those objects still intersecting - same problem with RAM and speed. The real problem is in legacymtl reflection glossiness.
what happens if you separate the object into individual planes, does this help?
Nope, I detached backside mirror surface, then scaled it down and even moved it back a bit - same problem. Check attachment - object on top is single sided glass and plane below is mirror.
Tried to explode surface into faces - same.
So now it looks like this: if ray passing through single sided surface with refractive and reflective legacymtl with reflection glossiness 0.95-0.99 (1) and without exiting media highly reflective surface(material) (2) reflects that ray back to the surface with the same legacytml material (1) - bug. Maybe ray should bounce between those surfaces (1) and (2) - I don't know.
Something like that.
what happens when you set max ray depth back to default?
RAM usage ok, didn't measured speed but visually looks like ok too.
P.S.
To make sure it wasn't a geometry issue - I replaced that "mirror" object with chamfers, which is editable mesh now, by freshly created box and the result is the same.