My advice would be to go with the Canon 15-85mm as your travel lens. It is so versatile, such a good focal range - and the IS and USM focus are amazing. I went the 15-85mm over the 17-55mm as the wider and more tele ends were important to me.
Also, when I want 'fast glasss' I go for primes. Most of my photography is outdoor (landscape, wildlife, macro and odd other pieces) I don't do much sports at all (apart from the occasional motorsports) - so I don't require a fast telezoom lens.
As neuro pointed out in his typically 'spot on' advice, having an ultrawide zoom, a 'good kit zoom' and a quality telezoom, plus a prime - makes a great travel package.
I will often take my Sigma 10-20mm, Canon 15-85mm, Canon 100mm f2.8 macro and Canon 70-300mm L as my 'four lens' kit. At times I'll take fewer lenses (eg 'just' 15-85mm and 70-300mm L, or 10-20mm and 15-85mm - depending on the types of photos I'll be taking).
If I just take 1 lens, it is nearly always the 15-85mm. The 17-55mm definitely is enjoyed by many others too, perhaps those that take more low light photos without a tripod (eg indoor portraits).
You listed the 100-300mmL as an option. While it has decent image quality, it's an old lens (without IS). I'd recommend the 70-300mm non L (which has quite ok image quality) above the 100-300mm L.