Love your ALL IN attitude!!
It is not that "ALL IN attitude" though.
As someone said, these are two different lenses.
From the moment you stepped into the prime world, your are doomed to choose, unless ready to buy both.
I don't own the ART but I rented the Canon version to see what is it all about. Now, since I discovered the art of 35mm I will have to find the budget for it; it was a big miss in my arsenal.
I use the 35mm when making a portrait more environmental. It allows you to go inside the event you are shooting like you are part of them. Example: dancing party, street manifestation, vacation portrait, life style, selfie, ...
The 85 is the beginning of telephoto. I use it when going close to my subject in order to focus on her/his feature (eyes, mouth, ...). Compression is the key word. The 35mm will exaggerate the distance between face features.
Using the 35mm for feature portrait becomes more like an art than portraiture.
I tried to use 50mm before but I hate it, it is in a no man's land. I like emphasizing what I am doing and as a result, I am fully aware of which lens I should use for what purpose; feature-based or environmental.
It is quite ok to use only one of them (I only discovered the 35mm a couple of months ago), but no one will help you choose your taste. Rent both and compare.