Full-frame cameras are expensive for two reasons: economies of scale, poor fabrication yields, and inefficient use of the wafer.
Economies of scale are straightforward. When you sell more of a product, the cost of tooling up the manufacturing line is amortized over a larger number of units, as is the cost of R&D. Additionally, if you sell more of a product, yield becomes more critical (because if you sell a million units, even a 1% failure rate is a *lot* of returns), which means that you're forced to do more work up front to polish the design, which tends to pay off in the long run.
Fabrication yields are also directly affected by the die size (the chunk of silicon that turns into a single chip). The larger the die size, the more problematic flaws in the silicon wafer become. Suppose you know that 10% of the surface of the chip will be unusable (it really isn't that high—I chose a big number to make the math easier). Suppose that manifests itself as a stripe across the middle of the wafer. If that wafer has a hundred chips on it, that might affect only the ones near the center—maybe 15 chips, which gives you an 85% yield—close to the 90% ideal yield. If that wafer has only four chips on it, that same stripe might affect two of them, resulting in only a 50% yield. If you're really unlucky, it might also affect the corners of the remaining two parts, resulting in a 0% yield.
Of course, realistically, a wafer that bad would be rejected, but there are lots of random flaws that can occur when manufacturing a chip, and the flaws almost all occur in rates that are proportional to the surface area. If you know that you'll have one bad pixel out of every million and a 10 MP crop body sensor, you'd expect 10 bad pixels on average, so if you consider 20 to be the bottom acceptable limit, most of your chips will pass. If you move up to a full-frame sensor with the same density, you're now at a 25.6 bad pixels on average, and most of your chips will fail.
Finally, there's the question of inefficient use of the wafer. The wafer is round, which means there's some waste whenever you make a square chip on it. The bigger the chip, the bigger the waste. Now if you're careful, you might be able to print some other minor parts (e.g. voltage regulators) in the remaining space so that you don't waste as much, but otherwise, there's a significant overhead to bigger dies.