Why is it always so fun to reminisce about a first camera?
Mine was a hand-me-down from my younger brother (does that make it a hand-me-up?) who decided to upgrade from a 400D to a 7D, and so I traded him my old iPod for the 400D. Used it with the crummy 75-300 non-IS non-USM, and a nifty fifty, until it had an unfortunate accident on a whitewater kayaking trip that resulted in my entire pelican box getting dumped in a river. It was enough camera to get me the bug, though, and led to buying a refurbed 7D and 70-200 2.8 , which has led to me having an entire shelf of bodies and lenses...
Disclaimer: I've never actually used any of the above lenses. I did just decide to upgrade from the 17-40 to the 16-35 f/4 IS, so I've read a LOT of reviews and comparisons the past few weeks.
"Is it worth it" is always a personal decision, so it's not our place to say whether the more expensive lenses are or are not worth the extra cost, for you. I will say, for non-professional use, I can't imagine the 16-35 f/2.8 being worth the price premium. There are professional landscape and travel photographers who rave about the f/4 IS lens. For that type of photography, you rarely, if ever, really need the f/2.8 aperture, and the IS is often more beneficial. Sharpness, color, contrast, distortion, flare... according to everything I've read the past few weeks, the f/4 IS is as good as it gets. IQ on the older versions of the 16-35 f/2.8 (v1 or v2) is definitely not as good, but v3 stepped up to pretty much match the f/4 IS.
The TS-E lenses and the 11-24 are somewhat their own leagues. 11mm is WAY wider than 16. I kind of see it being most suited to people who are used to ultra wide angles, and looking for that next step. I find that the EF-S 10-22 and 17-40 on FF still feel dramatically wide to me, which has kept me from spending a ton of money to go wider. The TS-E lens lineup is certainly intriguing, but they're pretty specialist lenses. If you're interested in "depth of field games" as just some fun and not too serious, the lensbaby lineup might be worth considering... they don't shift, but they do tilt, and you can have some fun for a lot less money (I got a whole kit for $80).
Also worth considering: Filters. Do you use them? The 11-24 doesn't take screw-in filters, although I believe someone has come up with a filter holder solution if you use square filters. The TS-E and 16-35 f/2.8 use 82mm filters. One of the selling points for me on the f/4 IS was that it's 77mm filters, so I can use the ND filters I already have for my other 77mm lenses (70-200, 24-105, etc). Again, I don't know if this matters to you at all, or if you already have 82mm filters, but it's something to think about.
Final thought: If you want wide angles, but don't know what lens to start with, I wouldn't start by buying a new 11-24 or TS-E. A used 17-40 can be had in good shape for $400, and I just bought my 16-35 f/4 IS for $630. At those prices, you can re-sell them for the same price you paid if you end up wanting the pricier lenses for whatever reason, and it ends up being a free test rental. But there's much more of a market for the less expensive options, so it'll be easier to resell them than it would be to resell an 11-24 if you start there and then decide it's wider than you need...
hope that helps!