I've just returned from the Olympics venues a few days ago.
I was at the Aquatic Center and at the Gymnastics (North Greenwich).
At the gymnastics stadium they search your bags (after the security search a-la pre-flight searches), and if you have a camera with an interchangeable lens that looks too big they don't allow it in.
They don't confiscate it, though, you can leave it in a locker and get it back after you leave.
They showed me the rules that state that you cannot enter a camera with an interchangeable lens that is greater than 35mm.
But they told me that up to 100mm is ok.
So my 70-200 F/2.8 IS had to be left there, but they allowed my 24-105 F4 (twice).
At the Aquatics center nobody checked my bags, but I didn't want to chance getting my 70-200 taken, so I brought the 24-105 and no one said anything.
If you're in one of the higher seats, I don't think anyone will notice you even if you have a 400 F4...
Downstairs, closer to the pool, they might, but your 135mm should be fine, as nobody said anything about my 24-105 even when I was at the AA seats very close to the pool.