The size and weight of all telephoto lenses is largely driven by their physical length, and the size of the front element. Those two factors are driven purely by focal length and aperture. For instance, get a 300/4, and with a simple bit of mathematics you can calculate the front element will be a quarter of 300mm - and the filter size is a very close match at 77mm. The 85/1.2 should have a theoretical 71mm front element, and it's filter size is 72mm. And so on.
This formula holds true for most lenses until the angle of view gets wider than about 45 degrees. At that point, the aperture is no longer dictates the size of the front element. Otherwise we would have a 14/2.8 with a 5mm diameter front element (entirely possible in a compact where the AoV is narrower than 45', but not possible on a FF DSLR).
It is with lenses wider than about 45 degrees AoV that they can be made smaller for these smaller imaging circles. Which is why normal zooms and ultrawides are typically made for crop sensor DSLR's, and no-one to my knowledge makes a crop only telephoto lens who's angle of view is (at its widest setting) much narrower than 45 degrees.