I'm puzzled by this too. The 50/1.8 has been literally the same glass since 1987. I had the MkI, pro-quality one. It and the 50/1.4 never were sharp, just the obvious combination of portable and low-light. I also had the 50/1.2 and again, just not sharp.
These (and the 1.0, also not sharp) are all double-Gauss designs. Then the RF50 and Otus 55 are utterly different, with no underlying design concept I can name by looking at it.
So I was thinking that perhaps improving substantially on the double-Gauss designs would make the lens so big that you lose it's portability aspect. (The RF50 is almost the exact size of the RF24-105/4IS), and so they stuck with portability.
However, Leica 50s have been quite sharp, and the APO-Summicron high-buck 50/2 is mentioned as the world's sharpest lens (along with the RF50 and the Otus 55) despite being a double-Gauss.
I may not represent enough buyers to make it worth producing, but I'd be happy to pay twice as much for a 50/1.4 or 50/1.8 that is a lot sharper but no bigger. That said, the Leice 35/1.4ASPH works fine on my R and I may just get another Leica lens for a portable 50mm solution.