As I understand it, modern lens design using new materials and CAD ray tracing techniques allows lens designers to achieve much greater resolving power, lower CA, better performance throughout a zoom range, etc. and in most cases the Mark II designs have shown significant improvements. It's not only about releasing a Mark II for marketing or competitive purposes, the new versions are typically much better and often add IS.
I've read posts from others on this site that Canon has a stated goal of improving lens resolution to resolve at least 40MP to keep up with DSLR resolutions. The resolution of FF sensors has now already exceeded that of film and the earlier generations of lenses just can not resolve the detail that the latest sensors are capable of. I know, I'm starting a film vs digital debate!