JMan, unfortunately, I do not own the 135L to compare it to, but I absolutely love my 100L. Like you mentioned, I use it primarily for portraits and only occasionally for true macro. Although I don't own it (I've used it once though), I would not trade my 100L for the 135L. They both offer amazing image quality, and I am constantly impressed with the results I get from the 100L. I think you would be hard pressed to tell the difference between the two lenses by comparing images from each.
Image stabilization, while not that important for the true macro work, it incredibly useful for portraits. It has been a great lens for typical portrait sessions, as well as low light wedding photography work. If I could choose just one lens, I would say the 100L is the way to go. I was going through the same decision about a year ago and decided the image quality and versatility of macro and IS were too good to pass up. Just my opinion of course.