Pixel size doesn't really have anything to do with it. A single pixel doesn't make a picture. Multiple pixels in concert make a picture.
Thank you, thank you, thank you.
I keep hearing "pixel size, pixel size!" and although I don't get the physics like you do, I can tell you that a single, full-frame sized pixel isn't going to take much of picture.