With the flagship RF camera now at 45 megapixels, I don't think you're going to see many new RF lenses that have a lot of compromise on image quality. What would be the point of making a lens that doesn't have the resolving capability to perform well on the flagship camera?
Like I said--compactness. The answer to your question is in the post you're responding to.
I always had the flagship EOS 1N, 1V, 1Ds 1DsMkII, MkIII... lots of lenses couldn't resolve at their pixel counts or even film resolution--the 1.4, 1.0, and 1.2 sure couldn't--but I was happy to use them anyways.
On my 7D, 700D and 350D.
I bought my 100mm f/2.8L and then sold my 100mm f/2.8 non-L (which was before I bought my 80D and M5).
So I can't vouch for the 100mm f/2.8 nonL's AF accuracy at non-macro ranges, using DPAF, or on the newer DSLRs.
However, I had repeatable experiences of 2 x copies of the 100mm non-L macro on multiple DSLRs, at non-macro ranges... it wasn't always accurate or reliable.
Whereas 2 copies of the 100mm L were much better to AF at non-macro ranges on all my DSLRs (i.e. accurate and reliable, as well as a bit faster too).
I expect the 100mm f/2.8 L will be superb on the EOS R5 (which is my plan for the future).