Interesting. Do you shoot at 24mm a lot? If you don't shoot at 24mm a whole lot, I'd suggest something that will surprise you. With your current kit I'd personally buy the 24-105L zoom lens. Then you can shoot wherever you want from 24 to 105mm whenever you want. If you really want a wider angle prime though, the 24L is really good.
Yes, surprising recommendation. I might consider the 24-70 f2.8 II as it would effectively replace what the 16-35 provided on my 40D, but I really do not like the barrel extension on lenses like the 24-70 or the 24-105.
Note: I also had the EFS 10-22 with the 40D, enjoyed it for awhile, but then rarely used it in place if the 16-35 (too wide).
Ok, so that narrowed your choices down. The 24-70L II barrel might be just as bad as the 24-105L. If it is then you know you don't want that, so the 24L choice is the right one in that case. If it's not it'll be hard to ignore its flexibility and IQ. As an aside, I shoot landscape and indoors with the 24L and it is fantastic. The 24-105L nor the 24-70L II will match it at 24mm in IQ, at any aperture. It is relatively small and light as well.