Before getting the EF 100-400 II, The EF 70-200 f/2.8 IS II was my most used lens. For the last couple years, the 100-400 II has been my most used lens: mainly for travel and for the kids' outdoor soccer games. When travelling, I pack the UWA, general purpose zoom, and the 100-400 II. A 70-200 f/2.8 is a lot better for indoor events.
On a recent long weekend trip, I brought the 15-35, 24-70 and 70-200. The compactness and the weight of the RF 70-200 was nice, but I still would have preferred having my 100-400 II. I would not like to choose between having only the 70-200 or the 100-400/500, but if I had to, I'd pick the 100-400/500 and pick up a prime at 85mm or 135mm or portraits and indoor events.