Thanks to all of you for comments - they're greatly appreciated!
To try to respond to a few of the comments and questions ...
I'm having a bit of trouble working out which focal length/s I prefer! For a long time I shot a crop sensor camera with 24mm (ie wide end of 24-70) as my widest option, which is close to 40mm in FF FOV terms, and I've only belatedly started to appreciate what I was missing out on in not having something wider. That said, I've done some playing around with my 24-70 on the 6D now and I'm starting to like 35mm. I'm finding I often feel like 50mm is either not wide enough or not long enough. In comparison, I feel like 35mm lets the photo "breath" a little bit more - wide enough to give a subject a bit more context, and the slight distortion (correct expression?) towards the corners adds a bit of interest to the photo.
I've thought about just keeping the 28 1.8. It may not be great but my feeling is it's not as bad as many of the reviews make out (or perhaps I got a good copy ... or I'm just not very discerning!). For the size and cost, I liked it a lot as a general purpose walk around lens on crop. On FF though I feel like I'm noticing its lack of resolution away from the centre more (well, I guess I'm seeing more away from the centre now), plus I feel like 28 is getting a bit wider than I'd like so I just don't see myself going out for a day with just the 28. Why is wider than I'd like? My plan is to use this lens (ie a lens around the 28mm to 40mm mark) for things like street photography, travel when I want to go light/inconspicuous, if I'm meeting friends (could well be at night) and want to be able to take some casual shots without the camera seeming too obtrusive. I feel like 28 starts to get a bit wide for people shots - it's fine if you have one or two people and you keep them close to the centre, but if you've got people towards the edges the distortion is likely to be too much. Anyway, I'm just not feeling excited about the 28 so it's not spending much time on my camera, so think I'd be better off selling it.
Given what I've said about the 50mm focal length, why not sell my Sigma 50 1.4, especially since it's not a compact lens? Well, I seem to have a fairly good one (I haven't had too much problem with its AF), I like the bokeh it can produce and the IQ generally, I doubt I'd get a whole lot if I sold it (used market here in Australia is a bit limited I think) and it's my only 1.4 aperture lens! And I'm not ready to give up on the 50mm focal length yet. I've seen plenty of great photos taken with a 50! I think it may be a case of me getting a clearer idea in my mind of when it's the "right" focal length for what I want to do.
As for the 40 pancake I really do like it. The IQ seems great to me, and it's small size and weight are fantastic. Even compared to the 28 1.8 (which I believe is slightly smaller than the 35 2 IS), the camera feels so much smaller and "easier" to carry with the 40. It feels like you're basically just carrying the body - there is (almost) nothing sticking off the front of it, so it's very easy to carry, squeeze into a bag, etc. On the other hand, somehow I feel like if I'm going to be limited to a single focal length, I should have a wider aperture than 2.8. The reality is for much of what I'd be shooting with this lens, 2.8 is probably enough - but I do like shallow DOF field shots and I'd really like to have the option to go wider than 2.8 even when I'm just wandering around. Also, I rarely shoot off a tripod so having IS would be valuable to me. It won't help with subject movement (to state the obvious) but I can see it being useful to me in many circumstances where I'd use this lens. I've only recently started to appreciate the value of IS in terms of being able to keep your ISO down and hence keep your IQ up in lower light.
So, I guess what I was really asking is - does having an extra stop on the 35, plus the IS, make enough of a difference that the 35 2 IS is a genuine upgrade from the pancake? Or does the 35 have better IQ than the pancake that it's an upgrade on that ground alone?
Of course, as was pointed out, the pancake is a pretty cheap lens to begin with, so I wouldn't get much for it if I sold it ... and it makes a very useful companion to a 70-200 (I do have one - it's focal length range I love on FF) so maybe it's worth holding on to anyway. That said, I see one comment that the 35 2 IS is small enough to fill the "pocket" lens role anyway.
All things for me to ponder. I think I'll see if I can find a 35 2 IS in a store and take a few shots so I can compare them to what I'm getting out of my 40 (and my 28).