The AF factor has kept me out of the lower end FF market.
When both of these cameras came out, I had the choice to buy either. I picked the 50D for the improved AF. The AF system on the 5D2 is the same as the 5D, which is the same as the 20D. I had issues with my 20D at times. I have fewer problems with the 50D. Ironic that the 7D came a year later with much better AF. ;-)
For my uses, I felt the IQ differences are minimal. Check this out: http://wyofoto.com/EOS_IQ_shootout_2008/EOS_shootout_2008.html
The only flaw with his comparison is using the same post processing. The cameras have different sensors... different signal processing... and require different settings in post. I took his sample files and did a little different sharpening on the 50D file... and got it to look as good at low ISO. I generally don't need high ISO, but with good NR I get clean images up to ISO 1600 easily, and I've found I can do a good job with 3200 as well... on the 50D. I haven't experimented a lot with 3200 yet.
Until you really push the print size and ISO together, I can't see any major differences in print.
For my uses, which does not *usually* include poster-sized prints, I have zero qualms with my prints from the 50D. I'm talking up to 13"x19". I came from 35mm film, and even the 20D blew my K64 and F100 slides away, so I just don't worry about it much.
Having said all that... their is one factor that would get me to buy a full-frame model... that is the desire to use certain lenses at the wide side, like the Zeiss 21mm. If you have a specific lens or set of lenses in mind, then FF might be the right choice for you.