This is a frustrating but expected result, as @Pokoy stated, the more cores a system has, the slower single threaded operations will be.
I vaguely remember an old CPU single performance chart from Tom's Hardware website.
With a list of many high end products starting from 1 core 1 thread Pentium 4 3gHz, this CPU was the slowest of all processors in multicore benchmark and number fastest in single threaded.
I also experience this a lot and makes me feel pessimistic about our profession. That's why I work with my 4 Cores laptop until it is rendering time. It is definitely more responsive during viewport operations than the 24 Core Xeon one.