It's called 'crop factor' or a 'crop sensor' for a reason - a smaller sensor crops the FoV, it doesn't magnify the image.
Comparing APS-C vs. FF cropped to the same FoV is going to be influenced by the specific sensors in question. But the IQ likely will not be significantly different (it's a wash between the 7D and 5DII, for example; I haven't done 7D vs. 1D X yet, but will soon). As for what you're giving up - MP. A 5DII/III image cropped to 7D framing is ~8 MP, not 18 MP. I have some 24x36" prints - 8 MP wouldn't be good for that.
The similar IQ applies in good light. If light is limiting, FF wins even when focal length limited, in terms of IQ. When not cropping, FF is the clear winner. For me, when I'm focal length-limited, I'm usually light-limited (need high ISO) as well. If my 7D vs. 1D X results are similar to privatebydesign's 7D vs. 1DsIII results, my 7D will likely get sold soon.