Take the longish zoom off the Canon and put on the Shorty McForty instead....
....
The only reason the 5D series has a reputation for being substantially bigger than a rangefinder is because all the comparisons are made with a normal slow prime on the rangefinder and a fast wide-to-telephoto zoom on the 5D. Mount similar lenses on each, and the size difference is more of a shape difference than anything else -- and the shape of the 5D makes it fit the hand better.
it´s a 35mm f2 on the leica and a 35mm f1.4 on the 5d mk2.
no longish zoom. 
Eh, sorry. Looked like a 24-105 at first.
But, again. An f/1.4 lens has an aperture twice the size as an f/2 lens, so is it any surprise that it's also substantially bigger?
Canon's own 35 f/2 is certainly no bigger than the Leica version, and looks to be a bit smaller. Of course, Canon's f/2 is a cheapie consumer lens while the f/1.4 is their no-holds-barred lens...but, if you want no-holds-barred quality in a small, light, cheap package, then that's what the Shorty McForty is all about.
a 35mm f2 would make things look a bit better for the 5D MK2 sure.. but not much.
walk a day with a leica and a day with a 5D MK2 through a city. 
it´s not only size.
the leica is less obtrusive (at least to the normal non-photographer crowd).
You do know that that type of distinction exists solely in your own head, don't you? Anything that isn't instantly recognizable as a P&S zoom or a camera phone is going to identify you as a photographer, and something that looks like a retro antique (the Leica) is going to make you stand out every bit at much as something big with a red ring that you hold up to your eye.
The 5DIII with the Shorty McForty, on the other hand, just looks like a slightly oversized PowerShot G-series-style camera, especially if you use Live View. It'll only draw attention from other photographers, not the general public.
Cheers,
b&