I find the ef-m adapter useful as a backup option, to extend reach, or to mount specialty lenses. Before I had the entire ef-m lens collection I tried all of my L glass on the M to see how they performed. I had used them on a T2i before so I was already comfortable with the crop factor. I appreciated the lenses with IS a bit more but found I could handhold them with appropriate shutter speeds.
I really prefer the native lens for their size and balance. In the photos you can see how I cradle the M and lens in my left hand, freeing my right for the controls. It's surprising how small and easy to hold is a 200 mm lens. It's somewhat similar to your golf grip. Lenses with the adapter were a bit more awkward.
My primary outdoor lenses for the M are the 11-22 and 55-200, with the 22 my choice for indoors. I find the 18-55 most useful for backyard video of the grandkid. The adapter still gets used for the Tamron 150-600, 100 f/2.8L IS Macro, and 8-15 f/4L fisheye which don't have ef-m counterparts.