The way your poll question is worded, the 24mm is a better answer. But reading your post, honestly, it sounds like the 35mm is the right choice for you. 24mm on FF is generally too wide for portraits - as you have found, it introduces quite a bit of distortion.
I went through a similar choice a while back, and I set my 24-105mm at 35mm for a couple of days, then at 24mm for a couple of days. When I processed the images, I found myself cropping almost all of the images at 24mm, and for the shots inside the house where there was a known framing of reference, it turned out I was cropping the 24mm shots to the approximate angle of view of the 35mm lens. So, I got the 35mm f/1.4L - it's been a great lens.
In addition to wide angle portraits (espeically indoors), I find that it's a great nighttime walkaround lens. Shooting with the aperture fairly open can produce some fun shots, too. Here are a few examples from a trip to China earlier this year:

EOS 5D Mark II, EF 35mm f/1.4
L USM, 1/30 s, f/1.4, ISO 100

EOS 5D Mark II, EF 35mm f/1.4
L USM, 1/50 s, f/2, ISO 200

EOS 5D Mark II, EF 35mm f/1.4
L USM, 1/250 s, f/2.5, ISO 1600

EOS 5D Mark II, EF 35mm f/1.4
L USM, 2.5 s, f/5.6, ISO 100