I wonder if offering numeric inputs wouldn't be good, along with a color swatch so you can either enter RGB values in 0-255 OR click the swatch to open the color picker and input there as you would now, whatever is more convenient to the user. So basically offer the same input for LDR values as you do for HDR linear.
Opening the picker, assigning the correct color space, choosing the color, pressing OK and still having to use the dropdown is quite a lot of mouse clicks for a single color assignment. I totally get why OP feels this is a bit too much for such a simple thing.