Devs do seem to look at this as a bug on some user's PCs, mine included but I come to same conclusion as you, other engines in Max don't behave this way, even stock Autodesk PhysicalMTL is pretty much instant, not super fast, instant, blink of eye.
Latest Threadrippers, Xeons, fastest laptop i9 HX,.. Windows 10 or Windows 11, Quadro or GeForce, Studio or Regular drivers. All the same. It's something in the design itself.