The composition of the video is quite good, as is the editing, but focus-pulling handheld is unheard of. Pro's use tripods and rigs with dedicated focus-pulling gears attached to the lens which they put markings on with a pen and rehearse before they shoot - you're attempting to to this on the fly!
Just set your maximum aperture to f/2.8 (shooting indoors or outdoors, never wider than this for video), your shutter speed to twice the frame rate and let ISO do the rest.
Another really important factor when shooting videos of human faces is to change the AF Mode (top option on 4th menu) from 'Live Mode' to ':-) Live Mode', it is the one that will allow you to AF on a human before you begin recording or at least get your initial focus correct before you start manually changing. For example, you could have gotten the skateboarder to perform a dry run and had him stop when he jumped off the board - focus on him then - then re-run the shot and he'll spring into focus.
Finally, horizontal panning in a DSLR should be done at a Snail's Pace (really slowly), only about one-tenth of the speed that you did in your video.
I think if you just get your key subjects in focus, you can add a lot of sharpening in post production (I do this all the time in Adobe Premiere Pro - you can even create unsharpen masks in PPro) and really tailor the look and feel of how you would like the final result to look.
Oh...buy a monopod, because it gives you all the benefits of a tripod, is quick to move around, better for panning (with a fluid video head) + it can be used like a boom for overhead shots, and you can run n' gun with it shortened