This is actually a pretty easy question to answer. You have a catalog of photos, and all of them have EXIF data that tell you exactly what focal lengths you used. So, go through (this is easy to do on a PC, or in Lightroom on Mac) and see what focal lengths you use most. If you're using the 24-70 in the 24-40 range a lot, then you'll be fine, as your 17-40 covers that. If you're using it a lot in the 50-70 range, then that tells you which prime you need to buy next, as a 50mm would be a necessity in your "all-prime" route.
As you see, everyone shoots differently and will have a different perspective, so the only real way to answer your question is to see how you shoot now.
