FOr what little it's worth, I've had good success with the 100-400 L II (whatever f/stop it is) on my R5, and, though I never owned a full frame DSLR to compare it to, I'm happy with it.
I was in the same boat as you actually, not wanting to get a 20MP camera. Other than that I'd probably have liked the R6 better; dual SD card slots and another dial instead of that little LCD display. Instead I have a CFE card slot I'll probably never use.
From what others have said, I see no point to going to a 5D IV instead unless your budget is tight.