If you shoot birds, by all means make the trade. For wildlife, it depends on where you are, 300+1.4x is enough for Africa, but not in other parts of the world depending on what you shoot. On the other hand, handholding a 500f4 on a boat shooting orca may be tough.
From what you describe, I'd get a crop body to go with the 300 and get the 2X III in case you need even more reach,
I loaned my 300f2.8 IS to my brother after getting a 500f4 II and never look back. I just came back from a trip to Africa and the 500 performed great for me from the safari vehicle. But I don't hike 4-12 miles with it and don't shoot orcas.