I recently returned from the second of two trips to Japan this year. Since these were both business trips, I wanted to get the maximum flexibility from my camera without having to lug around too much gear. In Japan I walk and use public transport a lot, and there is a limit to what I can carry alongside my luggage and my work laptop.
On the first trip I brought my 5D2 and the 24-105 kit lens only. That worked pretty well, being reasonably light and compact, but there were a number of times when I wanted more reach. So for the second trip I bought a new bag and packed the 24-105 along with my 70-200mm f/4L and a 1.4x teleconverter. This worked out great - the weight and bulk were not a problem and I got a lot of shots that would not have been worth taking with the 24-105 alone. I was shooting outdoors only, so the slow lenses were not a problem.
The bag is worth mentioning, as I think picking the right one makes a big difference to the experience. I got a Tamron UltraPro 7, a messenger style bag designed for a DSLR and two lenses plus accessories. I like the messenger style, compact for the capacity offered and it doesn't look that much like a typical camera bag so it attracts less attention. I also find the slim-line style puts less strain on my back than wider models. For trips when I bring more equipment I like Neuro's idea of bringing a large case and also packing a smaller bag to carry only the lenses I need on a specific outing. I plan to try this at the next opportunity.
As far as lens selection goes, the 17-40mm and 24-105mm should be a good combination. The 70-200mm f/2.8 may well be too heavy - I just gave in to my desire to own the mk2 version

, and it is certainly hefty if you are not used to big lenses. I am torn between selling my f/4 version to fund future purchases or keeping it for trips when the f/2.8 is too much. I need some time to get used to the 2.8 before I decide.
One comment about the 17-40mm - for architecture you need to be a bit careful, as it does introduce quite a lot of distortion especially at the wide end. I got this lens just before a Spanish vacation, and overused it shooting in cities not realizing how it would look. I was still using slide film at that time though, so this may not be as much of an issue with digital given the ability to correct in post. The ability to go wide in confined spaces in cities like London and Venice is certainly handy, since backing up to get everything in shot is often not an option.