I'm not really sure what to suggest, given what you've said. That said, I have some thoughts.
I have no experience with the Sigma version, but I do use the 70-200 f/4L non-IS. It's a great lens, dense color, excellent clarity, etc. throughout the range. However, I've had little success using it for street photography. It's generally too slow and too long for me. I'm planning to upgrade to the IS version this year.
You have the 40mm, I'm presuming it's the Canon f/2.8 pancake. I use this a lot, and it's very satisfying. It's brilliant sharp at f/2.8, it provides an almost ideal coverage for most street photography. However, I'm also wrestling with the notion of getting the Sigma 35mm f/1.4. I've seen a lot of great images from it, and it has superb reviews. I have a hard time justifying it though because how well the 40mm performs. But those extra two stops and the enhanced isolation capability are a strong sell.
Good luck. Let us know what you do.