Ha. This is exactly my situation as well. That S9 kit is remarkably capable for being so tiny - it might even be smaller than the R50V +kit lens. The weakness is that small primes for L are basically the Sigma contemporary series or manual focus.
I’ve been playing around with my wife’s R50 (not the V), and my feeling is that the ‘tiny everyday carry’ status is pretty fragile - the lens doesn’t have to be very big at all before it stops being ‘tiny’ and is just ‘smaller’. My concern is that a 20-50/4 with IS and PZ isn’t going to be small enough - if it’s around the size of the 16-28 STM or the LUMIX 20-60 it’s probably too big - not unusable, but no longer ‘stick in your coat pocket’. Still, if it’s small enough then the 16, 28 and 50 primes along with the 20-50 might be a pretty interesting travel setup. (I’d probably throw in the EF-S 55-250 IS STM for emergency reach assuming a crop mode works ok)