Have you mis typed this - you seem to be saying that 2 lens at f/4 will have different shutter speeds
I tested it out, and my assumption was incorrect.
At least in daylight photos, both my 16-35mm (@17mm) and 17mm TS-E at f/4 and ISO100 needed a 1/125sec shutter speed to get the exposure meter to come out the same. In very low light, I still thought I saw a difference between different lenses when using the same ISO and f-stops.
I did do one test in December that I should upload the pictures that showed the sharpness difference that you get by stepping down a 50mm f/1.2 lens in low light compared to 50mm lens with a slower f-stop rating.
And with the faster glass, you still have the option to drop the aperture in order to raise the shutter speed at the cost of depth of field.