Not at all. Every one of the 61 points is single orientation-sensitive (horizontal lines) at f/5.6. The middle three columns are f/5.6 crosses (with the central column of dual crosses also having a superimposed diagonal f/2.8 cross, which is what makes them dual crosses).
So, with most f/5.6 lenses, you have 21 cross-type points (middle 3 columns) and 40 lines. With most f/4 lenses, you have 41 cross-type points (all the orange and blue ones in the diagram) and 20 lines, and with most f/2.8 lenses you have 5 f/2.8 (more accurate) crosses, plus 36 other crosses (all the orange) plus 20 lines.
So, almost all f/5.6 and faster lenses use all 61 points, what varies is the number of those points that act as crosses vs. lines.
All 61 points work with all canon lenses f/5.6 or brighter
I'm using 'most' because there are some exceptions. For example, the 24-70mm activates only the center f/2.8 cross, not all five, despite being an f/2.8 lens, the 100mm f/2.8L macro lens doesn't activate any f/2.8 crosses (but still uses all 41 regular crosses + 20 lines). I say 'almost all use 61 point' because some very old zoom lenses, and the insanely expensive 800/5.6L IS, use only 47 total (the left/right sides drop off), and the 180L Macro only uses the central 33 of the 61 points.
Full details on which lenses activate which AF points can be found on this Google-translated page
That was probably way more than you wanted to know...