All my lens caps for my EF-M lenses (18-55mm, 11-22mm and 22mm) have a tendency to fall off from the lens or from my hand easily. I cannot even recall how many times that I have to pick them up from the ground. My new way to combat the problem is to use lens hood on all three lenses and not even use the lens cap. Lens cap will still be used when the "system " is in storage. As long as I keep the inside of my camera bag clean. It should not be a problem. I have enough room in my Nova 2 bag for each lens to be put inside the bag facing down. The lens cap also minimize the chance for my own finger to touch the front element of the lens by accident. I always have lens brush and blower in my camera bag. Any comment or suggestion is welcome.
Rocky, a fingerprint or dust on the front element of a lens will not appear in a photo, so don't worry about that. I do not put my lens cap on while my camera is out of its case.
I can't say why your lens caps are falling off, if you are using a filter, that might be a cause, but it sounds like you are not. I have little control over my fingers, and drop things easily. Because of this, I take a little extra time to insure that my lens caps actually went on properly and are securely snapped into place. Rarely, I don't properly secure it, and it drops off.
I hope you can figure that out. I actually use cheap center pinch ebay lens caps most of the time. They are a tad less secure than the Canon ones, but once I learned to check them carefully, they stay on.