I don't know much about the physics and science of optics, but the responses above sound pretty logical!
Also from a marketing standpoint, the pancake lens goes hand-in-hand with the smaller APS-C cameras, (ie an almost-pocketable DSLR!) and on a 1.6x crop camera, 50mm is too long for many people as general purpose lens. 40mm, on the other hand is a nice compromise between a standard view on both APS-C and FF.