I might be wrong but I think you just need the router to manage and assign IP addresses through DHCP. It doesn't need the internet. If you were to manually assign IP addresses to everything you might get away without a router. But it doesn't matter if the router is connected to the internet, it just needs to be plugged into your switch. So everything can still be on the switch, but the router needs to be on there unless you are going to do IP addresses manually (or if your switch is fancy and can do DHCP for you).
Hope that helps, that's my setup and it works like a charm. I did have similar issues to you prior to connecting my router.