The 100mm makes a good casual portrait / subject isolation lens for me too. The fast USM focus (especially on my 7D) is great. Some people say it focusses slow, but not mine!
I think the speed of the 100mm non-L is ok, too - but it takes about two seconds to focus completely from macro to infinity and back. In comparison to modern lenses (try the 17-55, 15-85, ...) this is slow like molasses.
Thanks Marsu42 for your reply to my comments above.

You're right that it might take two seconds if the focus is at the MFD (1:1) focus setting and the (portrait) subject is eg 3 metres away (in a poor contrast setting) and it cycles from macro to infinity and back. But what I mean is the focus is much much faster under 'normal focus composition' for portrait.
That is, if the lens' focus is set at 1 metre, and my (portrait) subject is 3 metres away, it will jump there in about a third to half a second (or less time in good contrast). That's why I also find the minimal focal distance (MFD) focus limiting switch on the 100mm macro handy.
Some people have reported that even in situations like the above, the Canon 100mm macro USM nonL is slow and their 100mm L is much quicker... but certainly mine isn't slow. My experience is that my 100mm USM lens's focus speed is very close to other Canon USM lenses when the focus is 'already in non macro settings'.
I have the Canon 15-85mm and the Canon 70-300mm L (perhaps my fastest focusing lenses) - and I also had the Canon 28-135mm and Canon 100-300mm (sold both these recently), and I'm very happy with the focussing speed and accuracy of all my Canon USM lenses.
On the other hand, I sold my Canon 50mm f/1.8 lens a few months ago - the main reason being the focus speed, accuracy and consistency were not up to my standards. I'm waiting for a true (full ring) USM prime from Canon in the 50mm - 60mm range, between f/1.4 - f2.
Cheers all, and thanks again Marsu42.

Paul