The 35 f/1.8 is very quick to focus, so not much gain there.
A USM L lens will definitely focus faster than an STM lens. Whether or not that makes a difference depends on the use. Environmental portraits, STM is fine. Sports, maybe not. Nano USM (assuming that's what is used) will also be quieter than STM, and that matters to video shooters.
Optical quality gain will also be small as the f/1.8 is very sharp.
The EF 35/1.4L II delivers substantially higher resolution (blue lines) than the RF 35/1.8, and better contrast (black lines) away from the center of the frame. Again, whether that makes a difference will depend on the use case, but I'd expect an RF 35L to be significantly sharper (even if the improvement over the EF MkII is minimal).
EF 35/1.4L II
RF 35/1.8
No argument on "build", but often a small light lens is less vulnerable than a big heavy lens even when not as robust. Also no argument re weather sealing for those who must do street photos at night in the rain, but for most, not a big issue for the majority of 35mm applications. Maybe the biggest issue is the overwhelming need for some to have "bigger and better" than the rest

.
Recent non-L lenses including the RF lenses are delivering very good IQ for a very reasonable price. The 100-400 and 28/2.8 are examples of lenses that punch well above their weight in terms of IQ for cost.
But the corresponding L-series lenses are optically better. Heavier and a lot more expensive, too. Some people just want the best. Some people can't afford the best, and for them it's great that they can get close for much less money. Personally, I prefer to use the right tool for the job and there are certainly times (e.g., travel) when the RF 100-400 that I bring will deliver infinitely better images than the RF 100-500L that I leave at home because it's too heavy.
Here's an example of a shot with the 100-400 that I took from near the summit of Mt. Etna, of an island that's ~90 km away. I would not have brought the 100-500L on the trip at all, but the lighter weight of the 100-400 made it worth packing for the few occasions I thought I'd need a long lens.
"
Salina"

EOS R8, RF 100-400mm f/5.6-8 IS USM @ 400mm, 1/400 s, f/8, ISO 1000