I have the 70-200mm f/2.8 L IS II and I would agree with many others that this is an excellent choice to widen your range of focal lengths. You already have the wide to mild tele range covered with your 24-70, so unless you want to go ultra-wide or super-tele this is a good pairing. I would go with the 24-70 II only if you are finding the mk I inadequate for your purposes, which you didn't mention.
I also had concerns about the weight after reading the reviews. Although I am not particularly big or brawny I haven't found it to be a problem. In large part I think this is down to the bag and strap I use. Based on advice from this forum I got a BlackRapid RS-4 strap, which I love. I also acquired a messenger style bag just big enough for my 5DII, 24-105mm f/4 L and the 70-200 (I think the 24-70 would fit fine in place of the 24-105). Along with a 1.4x III TC for greater reach, and an extension tube for occasional close-ups, this forms my "light" travel kit for trips when I am toting baggage and a computer around as well. The messenger style bag keeps the weight close to your body, reducing back strain, and the bag will also slip over the handle of a standard aircraft roll-aboard case for easy carrying in transit.
Once I am at my destination, I put the bag over one shoulder and the camera over the other. The BR strap allows the camera to hang upside-down at the waist, and the weight is balanced by the lens and accessories in the bag. I find I can walk or hike for quite some time without tiring or experiencing muscle strain, as I used to before I got this bag and strap setup.
As someone mentioned, the 70-200 f/4 L IS is also an excellent lens with significantly less weight and cost than the f/2.8 version. It is comparable in sharpness, has a slightly lower minimum focus distance, and is fine as long as you don't need the extra stop or narrower depth of field. For greater reach, there is also the 70-300 L to consider.