Another factor is weather sealing if you need to shoot in the rain - the 24L has it, the 35L doesn't.
Another reason im waiting for the 35L II! I get caught in the rain all the time with wedding photography. Ruined only 1 5d2 last year from rain. Stoked about the new 5D3 sealing. And a sealed 35 with a little better IQ than the current one would be a wonderful addition!