After I put a prime lens on my camera, I pretty much don't touch it until I take it off (I'd even say that applies to the EF 600/4 II, since I'm supporting it by the replacement RRS lens foot and not really touching the lens itself). About the only exception is manually focusing a lens for shooting the night sky (or lenses without AF, like the TS-Esand MP-E). Zoom lenses...I use the zoom ring, and that's it.
Well, apart from holding the camera from the lens (usually), it's not that I operate a lot of controls on the lenses either, but I do use their control rings sometimes, and I photograph exclusively handheld. I don't own lenses with custom buttons, neither VCMs with their aperture rings.
Also, I shoot absolutely everything using autofocus. Its smoothness can be felt in the hand and seen in the EVF. Plus, noise - or the absence of it.
External focusing designs, specially if with weird lens hoods (like the RF 35 1.8 and 28), are pretty much a vulnerability as they protrude from the lens housing. They're also more prone to allowing dust entering the lens - it doesn't take rubber gaskets for a lens to be a little more protected.
Refinement is subjective, but if you're looking for that then look for the L.
With the red ring often comes extra size, weight and, of course, the price tag.
I'm not asking a lot, I'm asking for attributes we had on gold and some silver ring EF lenses, like the EF 24 and 28mm f/2.8 IS USM, EF 35mm f/2 IS USM, EF 28mm f/1.8 USM, EF 85mm f/1.8 USM, EF 100mm f/2 USM, and probably others I'm not even aware of - all these were internal focusing and reasonably silent USM lenses with proper lens hoods.
So far, I think the RF 45mm is the only silver ring RF prime lens that is on par with that, in terms of "build quality".
Get me more lenses like that!

Perhaps a 16mm f/1.8, a 20mm f/1.8, a 28mm f/1.8, an 85mm f/1.8...or f/2, whatever, I'm not asking for f/1.4 or f/1.2.
Like I said, I would have paid an extra 100 or 200 bucks for the same lenses (RF 16, RF 28, maybe others) with a little more refined experience.
Yeah, not me. If I want my images to have a dreamy look to them, I'll just take a long nap...
I won't say I'm fond of the dreamy look because I'm not, but I don't usually shoot at f/1.2 either (my most used aperture is definitely f/2.8, across all lenses) so, for the price, I can live with that.
Ergonomics are great, autofocus is not annoying, I don't have to worry if something knocks it at the front, and a filter prevents dust from entering directly the lens barrel. As long as it's not raining and I'm not photographing in the dust, I'm good.