Have both but prefer the 135 for most any portrait work depending on the setting, distance. I find the 135 has a smoother bokeh than the 100. Don't really miss the IS but it would be nice.
Personally, I'm excited about what Sigma has to offer with a 135 f1.8 OS. I just rented the new Zeiss 135mm and will have that next week for testing.
I know it wasn't in your question but the 'older' 200mm f2.8L II makes a great portrait lens also, given you can step back 500 yards or so!

You really can't go wrong with either the 100 or 135. Both are sharp and consistent.
Best of luck.