Canon is and has been developing a radial diffractive (anomalous dispersion )optics that is basically a resin lens filled with minute particles, rather than the old design that uses multiple layers of gratings.
So far, they have been unable to come up with good enough process control to get a high yield. Those particles need to disperse perfectly to get the refractive index just right for each color. Since they continue to put out patents using the new technology, they must feel that its possible to make it work.
The patent 100-400mm design is 11.9 inches long at 400mm, the existing 100-400 is 10.91 inches(max). Apparently, this is a solution that allows for a very high quality lens, but its not smaller, its larger.