Tough call, they're both great lenses with unique attributes. The 300 f/4is is one of Canon's classic "sleeper" lenses. I regret selling mine when I got the 300 f/2.8is. It's just so light and compact in the bag, justifying a permanent spot in the bag. Most copies are pin sharp wide open. Mine responded well to the x1.4 extender, giving you a 420 f/5.6.
For what you have described, the 300 f/4is may be the best bet. It's a stop faster than the 70-300 f/4-5.6 at the long end for a start, plus you already have the 70-200 f/4.
If you can locate a pre-owned 300 f/4is you probably wouldn't lose a penny on it if you decided it was not right for you.