I shoot for a small college. I agree that volleyball is the hardest. To get faces you have to shoot through the net which makes autofocus next to impossible. It's hard to get shots of the players that are in the middle and if shooting from the side, the crowd can create a very distracting background. Plus the ball moves very fast, making it hard to get the ball in the shot.
I also find swimming difficult because so much of the action occurs underwater, so it can be challenging to get an interesting shot. Backstroke, breastroke and butterfly are not too bad, but with any sidestroke it is difficult to get a face shot.