Whether FF is an "upgrade" or not in the practical sense, depends on whether the user benefits from it. It's totally subjective, depending upon the needs and skills of the individual.
However, theoretically FF is always an "upgrade" since you are utilizing all of the light that can pass through a lens that fits the EF mount. This is the biggest deal for me. Also, technically if money is not a factor, you can achieve all crop benefits from a FF setup but not vice versa.
On the other hand, Canon provides a lot of alternatives to "make do" with a crop sensor while benefiting from additional features of the better APS-C bodies. There are excellent EF-S lenses and if you have fast primes, you can often get reasonably shallow DoF at equivalent FoVs unless you are looking for f/1.2 like Neuro said.
OP, have you tried the 15-85 before discarding the option as a walkaround zoom? Yes, it does not have the L ring or constant aperture but it is a splendid crop alternative to the 24-105. And the 17-55 deserves to be similarly priced as the 24-105. You will find the lens diagrams very similar, and the EF-S is a splendid lens. Although Canon should have included a hood, the cheapskates!