I find zooms to be more "convenient" for casual shooting. Primes are nice but against my set of lenses, I don't feel the need to buy every single L prime lens anymore. Lately, for casual shooting, I've been trying to discipline myself by carrying only 1 or 2 lenses at most and making it work with what I have on hand.
In short, yes, I say slim down your kit to what you absolutely need. It look like you can use an ultra wide angle lens in your kit. Also, the 24-105 might pair up nicely with the 100-400 instead of the 70-300.