It's not as if we're comparing L vs. non-L lenses. You can't go wrong with either lens, they're both excellent. The answer to your question boils down to what it's primary purpose will be and what your budget is.
If it's primarily for portraits, go for the 85L. DOF and focal length take precedence over minor quibbles such as the AF speed, it's not as if your subject is constantly moving.
Otherwise get the 135L. Too long for portraits though, but costs way less money and IQ is out of this world. Honestly, who would actually say that its F2 bokeh is crap compared to the 85L's F1.2?