I virtually never use a rectilinear ultra-wide anymore. I prefer my Sigma 15mm fisheye.
I've thought seriously about selling my 17-40L (which I consider to be a good lens, by the way) and I already gave away a 10-22 in favor of a Tokina 10-17 zoom fisheye. I've kept the 17-40L mostly because I already have it, and it's useful once in a great while.