Solved it. Use the Legacy material. Turns out this is an old issue I highlighted when the Physical Material was first released. I thought it had been fixed.
Physical Material - Diffuse 0 - Glossiness 1 - Refraction 1 - Thin Shell Unchecked = Problem
Physical Material - Diffuse 0 - Glossiness 1 - Refraction 1 - Thin Shell Checked = Problem
Legacy Material - Diffuse 0 - Reflection 1 - Refraction 1 - Thin (no refraction) Unchecked = Problem
Legacy Material - Diffuse 0 - Reflection 1 - Refraction 1 - Thin (no refraction) Checked - Problem Solved
So it seems that checking Thin Shell on the Physical Material isn't fully disabling refraction like the Legacy Material Thin (no refraction) does.
Edit: This is the original thread where I highlighted the weird behaviour of PhysicalMaterial Thin option back in 2021
https://forum.corona-renderer.com/index.php?topic=34168.msg189151#msg189151It seems the explanation was that "Thin Shell in the Physical Material now takes Refraction Overrides into account". The way I see it, as the tooltip states, Thin Shell should disable refraction. Any refraction overrides should also be ignored if refraction is effectively disabled.
And that's exactly the problem I'm having. The dummy geometry is visible to reflection/refraction because I need it visible to reflections. I don't want it visible to refractions though because I want to render on black. So I tick "Thin Shell" aka disable refractions. And theoretically that should solve the problem. The dummy objects would be visible to reflections still, but because the glass has refractions disabled (Thin Shell ticked), then the dummy geometry wouldn't be visible through the glass despite 'Visible to Reflections/Refractions being checked.