One other thing to mention, the larger the sensor, the fewer sensors yielded per silicon wafer. This, and the increased waste (wafers are round, so the larger the sensor, the larger the wasted fragments on the edges) translates into much higher cost for sensor production.
Someone feel free to correct me if I'm mistaken on this...
That's correct, but defects are equally concerning.
The reason it gets ridiculous really fast is you're pretty much guaranteed to have a certain number of defects per wafer (usually caused by dust), so in the hypothetical case that you get ten defects on a wafer but you're only producing really small sensors, all ten of those defects will only cause you to throw out a very small portion of the wafer.
If you were to try to make a sensor so big that you could only fit four on every wafer, at ten defects per wafer you'd pretty much be wasting your time as the chances of getting a single functional sensor out of it would be very low.
I remember reading a comment that Canon had to drastically improve air quality when moving to full frame production.