Since you are on a tight budget
Canon EF 70-300mm f/4-5.6 IS USM, is cheap and acceptable IQ, lighter and versitile, you can also try 2nd hand one too at about $4-500. IQ is probably similiar or marginally worse comparing to the 70-200f4L in the 70-200 range.
do not buy 70-200 IS f4 L now, most people end up selling that to buy the F2.8 one instead, is better to save enough money and buy straight off. Also your 24-105 is overlapping half of its range already.
as for the 70-300mm L , sure it is very good but cost twice as much as the non-L also heavier too. may be wait for the next 100-400?