Exactly my thoughts on the TS-E 24mm. I love shooting architecture -- which is the bread and butter of any tilt shift lens -- but with my son being born a couple weeks ago, I fear my architecture shooting days may be few and far between now.
I was also hesitant about purchasing such an expensive camera with a fixed prime lens. However, since my son was born I'm yet to break any zoom lens out of my bag. I've been exclusively using my 35mm, 50mm, 85mm, 100mm, and 135mm. I've been using my 85mm the most, as I really like the bokeh that lens produces as well as the the frame you get with the 85. BUT if I had to pick only one of these lenses and discard the rest, I would likely go with the 35mm for its versatility. So that made my decision on the x100s a little easier.
Congratulations on your baby! Is it your first? My life changed for the better, but with less time for self indulgence 
Thanks! Yes, my first. And life has certainly changed, and also for the better. I'm definitely taking more pictures (my 5Dm3 with a prime lens is always sitting on the kitchen table) and losing a lot of sleep, but no surprise with either of those.
When my wife and I found out we were expecting, I went on a prime lens buying spree. Literally picked up the Sigma 35mm f/1.4, Sigma 50mm f/1.4, Canon 85mm f/1.2L II, and Canon 135mm f/2L within the span of four months, and also threw the 24-70mm f/2.8L II in there for good measure.
So far I'm immensely happy with my decision to invest in primes, and also glad I have the 24-70mm ii for when my kid gets a little more mobile and I'll need some more versatility.
And also why I'm very happy with my decision to order the x100s over the weekend. A bulky SLR and primes are great for when we're around the house and I want to get some photos, but once we start traveling more with the baby, having something a little more mobile and easy to carry will be great. :-)