Exactly my thoughts on the TS-E 24mm. I love shooting architecture -- which is the bread and butter of any tilt shift lens -- but with my son being born a couple weeks ago, I fear my architecture shooting days may be few and far between now.
I was also hesitant about purchasing such an expensive camera with a fixed prime lens. However, since my son was born I'm yet to break any zoom lens out of my bag. I've been exclusively using my 35mm, 50mm, 85mm, 100mm, and 135mm. I've been using my 85mm the most, as I really like the bokeh that lens produces as well as the the frame you get with the 85. BUT if I had to pick only one of these lenses and discard the rest, I would likely go with the 35mm for its versatility. So that made my decision on the x100s a little easier.
Congratulations on your baby! Is it your first? My life changed for the better, but with less time for self indulgence 
Thanks! Yes, my first. And life has certainly changed, and also for the better. I'm definitely taking more pictures (my 5Dm3 with a prime lens is always sitting on the kitchen table) and losing a lot of sleep, but no surprise with either of those.
When my wife and I found out we were expecting, I went on a prime lens buying spree. Literally picked up the Sigma 35mm f/1.4, Sigma 50mm f/1.4, Canon 85mm f/1.2L II, and Canon 135mm f/2L within the span of four months, and also threw the 24-70mm f/2.8L II in there for good measure.
So far I'm immensely happy with my decision to invest in primes, and also glad I have the 24-70mm ii for when my kid gets a little more mobile and I'll need some more versatility.
And also why I'm very happy with my decision to order the x100s over the weekend. A bulky SLR and primes are great for when we're around the house and I want to get some photos, but once we start traveling more with the baby, having something a little more mobile and easy to carry will be great. :-)
The pub was suddenly a strange word, and sleeping in; not sure if I remember it at all

I see that you have the wonderful 100L in your prime lineup. It is wonderful for babypictures, at least when they are infants and somewhat calm in their movements. I took one of my second son (the same day that he was born. It is by no means a great shot, but I like it quite a bit). If you want, and have time to look at it you'll find it here:
http://500px.com/photo/2197306I will get the x100s in one way or the other. Whether it is after the selling of the 17 TS or by other means.
If I am not overstepping any boundaries, I would like to give you one advice on the newborn.. Don't mind the state of the house (there is a life left to clean), and take shifts sleeping. The latter is hard, because when the baby awakens it is natural for both to get up, but one will be worn out pretty quickly

Gerhard.