I have an S95 and it's a great little camera. I got it for very similar reasons - something that is truly pocketable yet maintains good IQ and can shoot in RAW.
I used to think there was never a reason not
to bring the dSLR and a bag of lenses. But last year, we went on a family trip up to New Hampshire, and I found myself with a 7D and a few lenses in a Lowepro PrimusAW on my back, going through the Polar Caves with passageways like Eye of the Needle and Lemon Squeeze, and nearly getting stuck...I ordered the S95 soon after getting home...
I have been on a lot of trips and carrying a big DSLR is a big downfall when it comes to weight. Airlines are not DSLR friendly. Going through the TSA is always a big hassle for me as they prefer to examine everything in my photography bag seperately for some reason.
Most of the time, I think it's worth bringing the dSLR. Yes, it's a challenge to carry a dSLR and a bag of lenses. I haven't had any problems with TSA or during international travel, though. I've checked the gear for a flight or two (although that entails some risk, it worked out fine - I was prepared in that I used a Lowepro Flipside 400AW for carrying my gear, and the loaded bag fits into a Storm im2500 hard case for travel - plus, the whole kit is insured). But even though I bring the 5DII on trips, the S95 comes too - it's good to have a backup, for one thing, and sometimes you just can't bring the camera backpack on an outing (at least when spelunking!).
There are two big downfalls of the Canon S95 in my opinion. 1) Battery life. I went to the St Patricks Day parade and after an hour or so of shooting with it the battery was nearly empty. 2) You are limited to direct flash since there is no provision for an external flash.
I haven't had an issue with battery life, but then I don't use it for video that often, and I carry a second battery just in case.
No provision for an external flash? It's compatible with the Canon High Power Flash HF-DC1
Ok, that's just a small, optically-slaved flash with a mounting bracket, but I mention it for the sake of technical accuracy.
Actually, I've found that between the f/2 lens and ISO 800 with the noise handled by DxO, I can do quite well in pretty low light - I almost never use the flash on the S95.