The enigmatic individuals who'd prefer to spend their money on a FF mirrorless rather than a 5Diii would only want small, high quality primes specifically designed for the body. They're not concerned with DSLR usability. Their driving desire would be unparalleled image quality.
This is a fair point, but, it begs the question of how many people like this their truly are. Because to get that IQ, you'd be talking a new body design, a new sensor, all new lens designs, and those costs have to be spread out among all potential buyers. Would Canon see it as worth it, making a system that's gonna cost as much as their 1-series cameras and cost nearly as much as Leica? Or is that more the role of niche companies to fill, especially if compatibility with EF/Canon equipment isn't crucial. Then again, it's probably not even as small a market as the $20k cine camera market that Canon just entered with the C300 and that series, so who knows.
Being able to use EF lenses on a mirrorless is I'd say a bigger issue since its questionable how large the market for dedicated quality mirrorless lenses is, you look somewhere like amazon and sales of everything but 1-2 pancakes and budget telezooms are very very low indeed compaired to SLR lenses.
To this, I'd agree that that audience isn't as large as say, the wedding photographer audience, etc. But, there are a lot of adventure photographers (think extreme sports, Nat Geo, etc) who would eat up the size of a mirrorless camera if it could match the performance of even the current 5dIII. Much easier to carry an NEX-7 and a pancake in the pocket of your ski jacket, or back of your kayak, then it is to carry a DSLR.
Of course, the demands of those activities are exactly where mirrorless is falling short at the moment (AF performance, handling of difficult light, etc). But when they figure it out, thats a target audience that won't turn back to DSLR's, because size and weight are critical on multi-day, and the only reason they are using DSLR's now is because its what works.
Again, this is a niche market though, so I'm not sure I'll see Canon as the first company to fill that need. Especially when Sony seems to be about 5 years ahead of them in that market