I was in a similar position, and I've rented both the 17 mm and 24 mm TS-E lenses. Although the 17mm gives some great perspective and is sharp corner-to-corner, I was not a huge fan of that bulbous element. Also, not being able to use screw-on filters was a major downside for me. The 24mm also worked quite well as a general walk around lens with my family, even if I had to rely on focus confirmation. So, I could take some landscape/architectural shots, and while walking around, still take family photos @ 24mm. The 17mm was purely for architecture/landscape with much less flexibility. For a family vacation, I would opt for the 24mm for its versatility.
I still have the 24mm TS-E on my to-buy list, but like a previous poster said, I'm also having trouble convincing myself that it's worth the $2k (granted the shift function was a great feature to have). Instead, I ended up dropping my photo budget on a HDSLR rig in hopes that I could be the next Tarantino with my family home videos (unfortunately, my videos will probably still be horrible).