There was no need whatsoever in the market to develop a 40 pancake.
Funny. If that were the case, then why is it doing so well in the market?
Because it is "cheap", optically good and most of all "oh so cute".
An equally improved 50/1.4 II at 399 or a 35/2.0 at 299 would also do very well. And an improved 50/1.8 II for 199 with improved IQ, 8 rounded aperture blades, Ring-USM, and metal mount.
Other than that: a 5D 1/2/3 shooter never looks like a P&S snapshooter, no matter what lens is attached. The body alone is too fat for that.
And it would be very difficult to take good photos holding a 5D 1/2/3 at arms length in live view. The body alone is too heavy for that.
Canon, like any other successful business, try to bring to market, something that they think can fill a hole in that market. People have been asking for pancake lenses, Canon produced one, which is optically good, "cheap" and "oh so cute" and customers lapped it up. Therefore they did it right and the marketing department did their jobs. A 50mm is the sort of lens that people get, because it is "standard", so they think they ought to have one, but how many would actually replace the one they had (whichever version) if a new one came out? I bet it would be less than those buying the shorty forty.
The reality is, I doubt Canon had the 5D in mind (if in fact hey had any body in mind) when they came up with the concept. Those looking for AF in video, wouldn't be serious film makers in the main, but those wanting to shoot a bit of home movie, the shorty forty is ideal for that I imagine, if on the 650D, plus they mostly wouldn't be interested in manual focusing (the focus ring looks a bit small). Also, it would be very unobtrusive and look like an expensive P&S on the xxxD line.