It's hard to tell exactly what's going on from your screenshots, but i think everything works as expected here. Layered material is blending sub-materials according to luminocity of its mask. If one or more of sub-materials is transparent, then layered material will be transparent too and not "stacked" like you're expecting. If you want that transparent material would reveal some other material, then you need to add additional geometry and assign it with that second material.
