I owned the 85/1.8 for six years, and then sold it after I bought the 100/2.8L IS. That was 4 years ago. My primary portrait work is headshots (head and shoulders) framing, and if anything, the 85's framing was a little too "loose" for my tastes, so I was always doing a lot of cropping.
I rarely shoot portraits at less than f/4, but that's because they're more commercial-oriented, rather than "artsy."
I've upgraded my 70-200/2.8L IS to the mark II version since buying the 100, but have never considered selling the prime.
The 85/1.8 is currently < $400, and you could probably recoup most of your purchase price, if you decided not to keep it. (After using mine for six years, I sold it for the original purchase price, but prices had gone up in the meantime, and mine was in like-new condition.)