Size, perhaps not, but unless I'm missing something, the weight difference could be considerable. The exit pupil would be the same size, and the length of the lens might be similar if it had the same number of elements, but all other things being equal, the diameter of the elements should be smaller because you need less glass to make a smaller image circle.
Now I know that the difference in lens diameter between EF and EF-S is not as extreme with longer lenses as it is for wide-angle lenses, but because the area (and thus, the volume) of a lens element falls off with the square of the diameter, even small changes could make a big difference in weight. For example, suppose an EF-S version of a 300mm f/2.4 were able to shave three millimeters off the diameter of the lens, from ~127mm to ~124mm. (That second number is entirely made up, BTW.) By my quick and dirty math, that tiny change in diameter would likely translate to almost a 5% reduction in lens element weight, which is not an insignificant difference.