I think you've hit the nail on the head.
Fixed aperture, especially fast fixed aperture zooms involve a lot of R&D to get right, and won't sell so many as simpler cheaper designs, so it's partly economics of scale.
The construction quality is almost always higher, more metal, metal mounts, often sealing, sometimes tripod rings etc.
They also tend to have more complex focus motor types (ring type manual over-ride USMs vs short throw micro-motors) and can often have more complex aperture designs with more blades etc.
I think there is an element (boom boom) of folk being willing to pay more for the best, and so they can charge more, but there is obvious physical differences in the bulk and weight of the lenses, they are quite different beasts.
There are some great budget lenses out there, like the 18-55 IS, the 55-250 IS (if you can live with the loss of light as you zoom) and the 40mm f2.8 etc, and the third party manufacturers do reasonably priced f2.8 zoom alternatives if you can't quite stretch to the canon versions.
It's not a stupid question at all, I hope I've helped, a bit at least.