I can attest to the quality of the 300 F4 L IS that you are thinking of buying. I sold my Canon 100-400 to pay for it. I find it to be a far more flexible lens for my uses. The closer minimum focus allied to the significantly better IQ made my 100-400 redundant. Also I found that the 300 F4 cropped to the same field of view as the 100-400 (at 400) gave better images.
I'd say mostly the opposite. I'm not sure how a 300mm prime is 'more flexible' than a 100-400mm zoom. I agree that the closer MFD of the 300mm is a benefit for flowers, butterflies, etc. I sold my 300/4L IS after getting the 100-400L - the latter delivers slightly lower IQ at the long end, but it's better than the 300/4 + 1.4x and better than the bare 300 cropped.
Good point - I should have stated that I used my 100-400 only at the long end. I find the 300 F4 L IS to be a superior and more flexible lens from 300 mm +.
My 300 F4 cropped (to 400mm equivalent) gave better IQ than my 100- 400 - this was the main reason I bought it! I did compare my 100-400 to a couple of others and it was as sharp, if not sharper, than those I tried it against so I don't think I had a soft copy, note I only tried it at 400mm.
I now have a 300 mm F2.8 L IS (1) and my F4 gives up VERY little to it IQ wise - perhaps I just had a particularly good copy of the 300 F4, though a friend of mine has one that performs the same.