Anyone have any thoughts on this crazy idea?
Depends on what is most important to you. If you find yourself needing the extra reach and the 10FPS, then it might make sense. However, for landscapes and portraits, I for one would want a full frame camera. Also, it would be good for you to go to a local shop that might have a 1D IV to try out. I for one think that once you get used to the menu system and the AF layout of the 5D III, going to a 1D IV might seem awkward and a step back.
6FP is not all that bad. I have captured some amazing wildlife and sporting shots with a 5D III. I have a 1Dx now, and soon to have 6D. But the 5D III is a wonderful camera! If the buffering is the only issue, you might invest in a new CF card, a Lexar Professional 1000x 32GB or higher (the 16GB are artificially slowed down). Also take out the SD card that you might have in the 5D III. I believe leaving an SD in causes the bus to slow down even for the high speed CF card. Give that a go and see if you need something more.