Respectfully disagree on a couple of assumptions you made.
I am willing to bet that the mk.ii costs less than the mk.i to manufacture. (Cost to manufacture, not fully loaded Overheads), since productivity in process manufacturing helps out a lot here. Plastic vs. metal machining etc.
The mk.ii is not pricier, since both were made in Japan, the local price in Yen is almost the same as the mk.1 was 10 years ago, taking inflation into consideration, it is actually cheaper than the mk.1. The price increase in US$ is a result of the devaluation of the dollar more than anything else.. not becuase making mk.ii was more expensive.
Price is determined by what the market can bear also, not only the price to manufacture. If no product comes close to the 24-70 in performance, then a premium will be assigned to it... and the market will pay, if not then you will see a drop in price... yet they will still sell at a profit.
I am willing to bet also that the price to manufacture a 24-105 f4 might be close or even higher than the 24-70 ii on a purely material and assembly cost basis. Is Canon losing money on either? No... it is just loading overheads differently to each product... This fuzzy accounting is what I deal with everyday catering to both Finance and marketing as I provide solutions to both of them to manage profitability.
At the end it depends how R&D costs are spread over products... different companies do it differently..
To surmise, the improvement in quality is due to newer design technologies available, not because they sprinkled magic fairy dust into mk.ii..
I respectfully disagree with almost all of your assumptions. I'm willing to bet the opposite of all of the things you mentioned you're willing to bet. Specifically, I'd bet the new 24-70 is significantly costlier to manufacture than the old one, and that it's significantly costlier to manufacture than the 24-105. You really think they would make this product significantly better in almost every way, but with
lower production costs? That would require some magic fairy dust indeed — magic fairy dust that boosts worker productivity, lowers material costs, improves quality control, provides outstanding engineering, etc., ... all for less money. No doubt newer design technologies help, but newer design technologies don't suddenly appear for free. A manufacturer typically has to make a significant investment in order to make newer design technologies pay off. Building or re-equiping a factory isn't exactly cheap.
You may deal with fuzzy accounting everyday, but I don't think you're in the camera & lens business (and neither am I). You're probably correct that they load overheads [somewhat] differently to each product, but I suspect the differences are not very dramatic and that the ultimate selling price has a fairly predictable relationship with the manufacturing cost.
Canon's new 24-70 is 18% or so more expensive than Nikon's 24-70, and that may be partly due to it being new (and thus having premium for early users) but the price will likely drop 10% or so in the next year and then it will be closer in price to Nikon's — and then it will not seem so extreme. Nikon users have long been able to say (correctly) that their 24-70 offers better in sharpness and reliability than Canon's, but that never surprised me as Nikon's was about $500 more expensive. Of course, that extra $500 bought
something. Now, it is evident that the extra $1,000 in the new 24-70 (vs. the old one) also buys
something ... namely, quality and durability. I really don't understand how people believe that a lens manufacturer will deliver a significantly better lens but at the same or lower cost to manufacture, and with a massive jackup in selling price just because they can. I suppose it can happen, but it's not very likely.