When it comes to lenses Jean Reno said it best in the movie Leon;
"The tele is the first lens you learn how to use, because it lets you keep your distance from the client. The closer you get to being a pro, the closer you can get to the client. The super wide angle, for example, is the last thing you learn. "
Ok so perhaps that was not the exact quote...
While focal lengths of about 80-135 will give the most flattering look on your subject, a wide angle combined with a lot of energy can give great results. But be aware of a few things - distortion gets worse the closer to the frame you get. So place your portrait subject in the centre if you want the face to dominate the picture. I think the best way to use a wideangle lens for portraits is to do full body portraits with lots of depth of field. When the head is smaller in the picture its harder to spot if the nose seems a bit too big.
So, CAN you do great portraits with a 16-35? Yes. But it is hard to make them look good. Does it really add anything to your 24-105? Not reall, for portraits. And the 24-105 @ 105 /4 is not too bad when it comes to portraits. So when it comes to your portraits, use the 24-105 most of the time and then go crazy with the 16-35 and try to build up loads of energy - then you really don´t care as much if things looks a bit off.