When wandering Rome, you are correct the 70-200 isn't going to cut it.
Personally I almost never take that lens off (and yes, I'm using a crop body, not FF), but I neither could I shoot a landscape or architecture to save my life so I don't usally even try. But even with my strict focus on people and sports there have been a few instances that I couldn't back far enough away to get the shot with the 70mm minimum. I love the lens and use it 95% of the time, but it is not going to have a ton of flexibility if you want to move beyond head-shot portraits and sports (and maybe street? Dunno, never tried...)
When I need to go shorter, the 24-105L produces great images, so if you don't need the speed or short DOF, I can easily recommend it. For my uses either I or my target seem to always be on the move, so a zoom is almost indespensible for me and as much as I love the IQ of primes they rarely make it onto my camera.
Look at the EXIF of the photos you've shot and figure out if you zoom a lot or kinda keep it at one spot, and shop accordingly.