Thanks for the replies.
I'm a hobby shooter and most of my shooting for the past one and a half year was with the 24-105 f/4, 70-200 f/4 and the 100-400 at f/5.6. Personally, I found these lenses wide open (at f/4 and f/5.6) to give me all the DOF I needed and probably more which got me into getting the faster lenses.
The 24-70 II was purchased earlier this year and it has been used between f/2.8 to f/5.6 but more often than not, wide open.
The 50mm f/1.4 prime is used mostly indoors in low light where I need the lens faster than f/2.8 - so again wide open or f/2.2.
The 135L is used when I'm taking pictures of my kids / in low light and/or while shooting random sports - so mostly at f/2 to give me great bokeh and fast shutter speed.
The only lens I use mostly stopped down is the 100mm macro which I found I use mostly at f/11 or smaller.
The 16-35 II and the 70-200 II were acquired just this past week so things may change yet