Neuro, have you used the 400 f2.8 and the 500 f4 and if so, why is the 400 roughly 4 pounds heavier? Is that 2.8 glass that much of a difference? The 500's are BIG and heavy, we'd like to get 400's but the weight....phew! Not to mention, the price...
400/2.8 = 143mm diameter front element, vs. 500/4 = 125mm. A 14.4% larger diameter means a 31% larger area, and since elements aren't flat, larger elements are relatively thicker, too. So yes, it's the glass.
OTOH, the new 600/4 II and 400/2.8 II had some dramatic weight reductions, they're basically the same weight as the 500/4 MkI (the 400 II is 20 g lighter, the 600 II is 50 g heavier). That's thanks to fewer elements (including elimination of the protective meniscus lens in front of the front optical element) and more titanium in the construction. The 500/4 II shaved 1.5 lbs off the MkI (decent, but much less dramatic than the others - the 500 lost 8% weight, the 600 lost 28%). Of course, the cost went up. A lot. Bottom line, if you can handhold a 500/4 MkI, you can handhold the 400/2.8 II and 600/4 II.