It worked for me, on a tripod, shooting skyline shots of Miami the first time. I had the filter off, it ghosted. I put it on, it didn't, or didn't nearly as much. I pointed it at other light sources over the 3-4 years I had the lens with and without the filter, and every time, the filter helped suppress the ghosts.
I'm not an optical engineer by any means, but I believe it works because it lessens the amount of the light from bouncing off the sensor from bouncing back into the sensor, which is what's causing the ghosting. The reflection off the sensor is going to hit the foremost piece of glass and reflect back into the sensor. By putting a better coated piece of glass in the front, I think it reduces the ghosting.
Yes, with that and some other older lens designs that were pre-digital. I know it seems backwards, but if you think though it, the reflections are coming off the sensor (not an issue with film) and reflecting back on the front element. If the filter cuts down on the (internal) reflection with a modern multi-coating, that explains it. Again, it's no good in bright light, but helps with this specific type of reflection on lower intensity lights.
I found that my 50 1.4 did this same thing a lot, so I put a high quality multi-coated filter (like Hoya or B+W) and it took care of it. It won't fix flare from the sun or really bright lights, but should work in the example you posted.
You find you get less flare with a filter on?
Actually, physically this makes no sense. A filter in front of the lens will not prevent light bouncing off the sensor from reflecting from the front element or any other internal element. Once the light is through the filter, nothing can prevent any internal reflections.
That may or may not be sound logic, all I can say is that when I got my 50 1.4, I couldn't believe how badly it ghosted with night shots very similar to the OP's. I screwed a multi-coated filter on the front and the problem practically disappeared. Going back in my mind, it was a 450D, 50 f/1.4, and a Hoya HMC Super filter.
Ok, so how is a filter (let's say it lets 95% of the light through) supposed to lessen the amount of light bouncing off the sensor? The only filter that can reduce the incoming light is a neutral density filter, but it just makes all darker if you didn't adjust exposure (which would actually reduce reflections because light intensity is less)
Regarding your 50mm f1.4 experience, the most likely explanation is that you did not do it under controlled circumstances, i.e. the shots with and without filter were not identical. You can only compare if you put it on a tripod and do with and without filter.
I won't debate this any further. It worked. Period.
If it really worked, I would be very interested to see a sample.