@Mackguyver: great run down on the lenses, but according to TDP and my own experience the 35L is at least equivalent to the 24-70 II at f/2.8 and at wider apertures, the lack of sharpness might actually be due to the CA.
@OP:
I have all the three focal lengths, and for me the 35mm is just an amazing versatile FL that works for everything from landscape to street photography to portraits. There is shallow enough DoF and great bokeh at f/1.4, practically no distortion, and it is tack sharp. Focuses pretty quickly and accurately, and is quite compact. If you want to save $$ and need IS (like you, I have nothing with IS below 70mm) then you might want to go with the 35/2, but for me the f/1.4 was more important. I needed something for those truly dark times...

The Sigma is a great lens and I loved it during the short time I owned it, but I am not willing to deal with AF inconsistencies. I don't think that additional bit of sharpness will make me a great photographer, but images with missed focus (esp. those I can't detect in the LCD) are lost forever.
The 85mm is a perfect portrait (1/4-1/2 body) lens. I just bought the f/1.8 with a lot of hesitation as whoever compliments it, qualifies it with ("for the price"). However, it is a pretty good lens. It is as sharp as you NEED wide open, and has really fast AF and is light and compact. I will probably sell it soon after I don't need it anymore (right now I don't have access to my f/2.8 zooms) as it is only 1 stop faster than my 70-200. However, you will greatly benefit from the faster AF (the accuracy of multiple AF points in your camera are much faster with a >f/2.8 lens). It is also a great candid street lens, as I think a longer focal length with great subject isolation helps me most in that situation.
The 135mm is my most favorite lens. It travels with me everywhere. Amazing whenever I use it. As Mackguyver said, absolutely no problem hand holding. However, in your case as so many have said, the 100L probably makes more sense (for me, it doesn't offer much more than macro in relation to the 70-200). It is also close enough to 85mm that it can do double duty for portraits. And it is not too slow to AF, with the AF limiter on.
So, in your place I would probably go with a 35mm and the 100mm macro.