I can't help but feel that the biggest downside, in many peoples minds, to MILC is that their investment in glass would be obsolete without adapter.
That is one of the issues. But I think the real issue for most people that are hesitant (and people that have tried them) is overall usability.
Let's be real here. AF still blows (speed and consistency) on every mirrorless offering in the market relative to a DSLR. The only piece of tech in existence currently is Canon's dual pixel tech which can potentially be used to provide something that can bring mirrorless AF up to par (or perhaps beyond a DSLR).
Yes, there are great manual aids that various companies provide in their mirrorless offerings. But the average user is the one that companies need to pursuade as they are the largest percentage of consumers and most of them don't want to have to manually focus. Also, focus peaking isn't all that great when it comes to super fast lenses and getting consistent critical focus. Split prism and/or zoom PIP is okay, but still not very fast or usable for all situations.
The second issue would be energy consumption. Battery life sucks on mirrorless cameras. Yes, all of them. I try to use the OVF and just deal with parallax as much as possible on my x100s and it is still not getting great battery life. Although I can deal with it, I still don't like having extra batteries in my pockets when I'm running out the door with the family in a hurry. So unless there are some earth-shattering developments in battery tech in the near future, this will be a major issue for the average user with any mirrorless camera.
Then comes the lens issue for me. lol. That being said, if the two problems I mentioned above were somehow rectified by some miracle, I would have no qualms about dumping my glass and going all in on a mirrorless system.