Simple solution! Don't photograph the family that way... Just shoot them in their natural environment doing what they like best... You are more likely to get keepers that way - but then you knew that already
I do... all the time... because that is the only way I get natural photos... but it is the group photos that are requested. I'm like a rap DJ at a hot NY club who is asked to play some Sweet Home Alabama... sure... I love me some Sweet Home Alabama... but will THIS crowd dance to the tune. Nope.
Maybe there's a different way.
It seems that you get acceptable shots once the subjects are comfortable with your presence as you shoot them one-on-one, or in smaller groups, and that the problem really arises when you have a large group, who probably get restless while you compose and shoot.
Try something that turns the "oh, no, a group photo" into a "yippee" moment.
Make it fun! (Now I don't know what that will take in your circumstances - to make it fun long enough to keep their attention.
If the young ones are the troublemakers - give them something fun to do as part of the shoot!
Just an idea.