While not knowing your financial situation, I can recommend keeping the 40mm. It works well as compact body cap that still has your camera in a ready-condition. Additionally, you may always find situations where you don't want to risk your nice 35mm to the elements you're shooting in. Far better to risk a $150 lens in those instances. Not to mention that it is relatively similar to the 35 and 50 focal lengths to serve as a useful backup in case the unthinkable happens to your primary lenses. Similar to why I kept my Mk I 50mm f/1.8 loooooong after I got the 50mm f/1.4, and still keep it despite also owning the 40mm. They were cheap investments and easy to justify keeping around for those "oh crap" situations. I got my 40mm for $130 open-box from Best Buy when I wandered past their photo cabinet on my way to get a TV. I instantly loved it, despite the fact that I really prefer the 35mm focal length as a matter of choice in shooting.
I have a whole litter of lenses, but the one that is on the camera I always have near me "just-in-case" is the 40mm. Small, unobtrusive, and more than capable of capturing fleeting moments in most documentary situations. The 40mm, a 6D, and a 580EX II now form the cornerstone of my lightweight running photography pack. Sometimes I go out with nothing but that setup and the 135L in my bag when I'm cycling or running with a group or for long treks away from roads during ultra marathons, etc.. Carrying the 24-70 zoom or even one of my 35 1.4s would significantly impact my range given pack space, weight, and food considerations.
Your situations may not be similar, but consider those situations in your own photography where being light, small, and unobtrusive would be a plus. Not every situation requires IS or the best IQ you have available in your kit at or near a given focal length.
It's a cheap investment that you've already made.