I just sold my 24-70 II last week...i found it a good, sharp lens...but also kind of boring. With my prime Lenses i can get more pictures that just "pops" out.
For me...the primes are the way to go 
I don't have hands on experience, but am debating the same thing. I just upgraded to the 5DIII and currently have the 24-105 f/4. I am plenty impressed with the 24-105, but I am even more impressed with the comments and reviews I have seen of hte 24-70 II. So I have been debating, sometime in the next year, do I sell the 24-105 and buy the 24-70 II or do I start collecting primes. I am also making the same debate in the 70-200 mm range (70-200 f/2.8 II vs primes at 85, 135 and 200).
I am coming down on DOF. The DOF of f/2.8 at 10 ft is 11.1 ft @ 24 mm, 4.4 ft @ 35 mm, 2.1 ft @ 50 mm, 1 ft @ 70 mm, 0.7 ft @ 85 mm, and 0.27 ft at 135 mm. Compare to f/1.4 at 10 feet focus distance and you get a total depth of field of 4.7 ft at 24 mm, 2.1 ft at 35 mm, 1 ft at 50 mm, 0.5 ft at 70 mm, 0.35 ft at 85 mm, and 0.14 ft at 135 mm (granted, the 135 is f/2, which has a DOF of 0.2 ft).
So from 70-200 mm, I am currently thinking I will get the 70-200 f/2.8 over primes, eventually, as f/2.8 gives me most of the DOF I will need (I am a hobbiest, not a pro). But you start getting down in the 24-35 mm range, and I can see how the DOF would limit your "artistic" ability. If I had to buy my kit right now, I'd probably keep my 24-105 as a walk around lens and buy primes at 24 mm, and 35 mm.
I am coming to this decision a little reluctantly as I have seen so much love for the 135 f/2 and the 24-70 f/2.8 II. So I may still change my opinion, but I am currently thinking those may be luxuries for much further down the road...for me, that is.