If you're used to and happy enough with the SL1, get the SL2. It's the same, just better in every single regard. An SL2 with the EF-S 24mm & EF 40mm 'pancake' lenses works wonderfully for environmental and looser-cropped portraits. Obviously if you want to do anything tighter or with more variation in depth of field then you'll need a longer and faster lens, but it sounds like you already have that covered. There are also a lot of compact APS-C zooms these days which are much better quality than the old 'kit' zooms, which pair very nicely with the SL2. (The Sigma 17-70mm f/2.8-4 compares pretty well to the 24-105 f/4 L, considering how much cheaper, smaller, and lighter it is.) Of course if you already have a lot of lenses which will work well for you then that's another tick in favour of the SL2; the full-size Canon lenses are really awkward and unbalanced on the M5, and while the SL2 isn't a perfect match for them, it's at least a little closer with slightly more surface to grip.
One thing to remember is the SL2 can be used like a mirrorless camera or like an SLR, whereas the M5 is 'stuck' as a mirrorless system. Now, I don't think mirrorless is bad at all—these days all of my personal shooting is done with Fuji mirrorless cameras, while my Canons have been relegated to work only—but it does still have significant limitations and quirks (more battery drain, AF trades speed for accuracy, etc) which won't suit everyone. The SL2 offering both shooting styles in one body makes it really very ideal as a backup. You can get all the advantages of mirrorless by using the SL2 with live view when you can afford it, and when you need to save battery or focus that little bit quicker or if you just prefer using an optical viewfinder, you've got that option.
You're unlikely to ever end up using the SL2 primarily (or the M5, for that matter) over any other cameras you already have, but as a space-saving "2nd body for portraits", it's pretty ideal. The only cameras which provide better functionality and quality are either bigger (35mm sensor Canons) or would require whole new lens kits (Fuji, Sony, etc).