On a production scene, with exact operators and values as you have, and with 4000x4000 resolution:
E.g. contrast operator turning on/off is not as fast as with lower resolution, but having curves enabled or disabled, with histogram, did not result in any change.
So this is either scene or hardware related.
Can you please tests and see if there is slowdown with lower resolutions, and with the default operator stack and then curves at the end?
Please try with VFB 1.0 too.
If there will still be a slowdown, I think we will need the scene to investigate it further. But I am not aware of many "slow VFB 2.0" reports.