I have the M and have used it a few times. I have both the 22mm and the 18-55mm lenses. No adapter (tried it and the focus with standard EF lenses was far too slow for my liking)
Unlike many posters here, I didn't buy it so much as a replacement or even complement to my DSLR as I did get it to replace my SD950IS Elph. I bought it as a small everyday camera and use it for the same things I would have used the Elph for (read: places where a DSLR is either inconvenient or not allowed). FOR THAT PURPOSE it excels. The IQ runs circles around the SD950IS and (using the 22mm lens) rivals my old 60D in quality. The two big drawbacks I see are a) lack of built-in flash and b) AF can hunt sometimes. The AF is no worse than the SD950IS in most cases (sometimes it is slower at wide angle, but at telephoto it's significantly faster - the SD950IS would take at least 2-3 seconds to focus zoomed all the way in).
The 22mm lens provides great IQ - comparable to, say a 60D with the 40mm 2.8. OTOH, the 18-55mm I find to be very average, even mediocre in non-ideal lighting situations. Even then it's STILL better than the SD950IS. Disappointingly, though, sometimes there's not much difference (usually shooting ISO 3200 indoors/no flash).
But the biggest upgrade from the Elph for me? The ability to shoot RAW...this keeps my workflow consistent and gives me the same options/flexibility for editing as if I used my 5D.
As a P&S replacement - get it, it won't let you down. As a DSLR replacement - not recommended. As a complement/backup to a DSLR - perhaps I would wait and try out the new Rebel SL1 first.