A few thoughts that echo what others have said.
Technically, the shots are excellent.
Group shots are hard. Getting one person to look comfortable and relaxed isn't easy. Lots of times, two people can actually be easier because they can play off one another. But, three to four gets really complicated.
That's especially true with family groups and even more so with families with older kids. Anyone who has ever been through puberty knows what a painful time it can be and how conflicted kids are in their relationships with their parents. It's just a tricky time in the child's and parents' lives. Also, in this case, the father seems very uncomfortable in front of the camera. There seems to be a lot of stress in the portrait session and the strained relationships show through.
I'm not sure what, if anything, you could do about it.
One thing I think is very important is allotting sufficient time for a portrait session. You need some time to get everyone comfortable with having their pictures taken. Sometimes you'll get really lucky and you can get a great shot in the first few minutes of a shoot. But, more likely, you need to burn through a couple hundred frames before the subjects get comfortable and pliable.
Sometimes, no matter what you do, the formal portrait just isn't going to work. That's when it's time to head to a park and try candid shots of them just interacting with one another. A nice spot in some open shade and you can forget about lighting and focus on the subjects.
Ultimately, portraiture has nothing to do with lighting and everything to do with relationships. The relationship between the subject and the photographer and the relationship between the subjects in a group portrait. This kind of group portrait is a difficult assignment and for a first effort with a very challenging group, you did a great job.
Having said all that, here are a couple of technical tips.
First, the shot of the couple with the dog is great. The couple looks relaxed and the dog is very natural. Super hard to get.
Try breaking the plane. In the group shot, everyone is pretty much positioned in the same plane. Try mixing it up a bit, arranging them so that they are in different spots in relation to the film plane -- someone closer, someone more distant, less of a "line them up and shoot them" look. (This requires a smaller aperture to keep everyone's eyes in focus and greater distance to the backdrop, so that it remains soft or indistinct, but it can be more interesting.
Get off of eye level. Stand on a stool and look down on them, have them cock their heads up. This makes the head the largest object and closest to the lens, which is generally more appealing.
Find a prop. Take a small loveseat or couch and arrange them around it. Some seated, some standing. Shoot from slightly above to add some visual interest. Props also give people something to do with their hands. Use something timeless though (a soccer ball for instance, if the kids are into sports). You don't want to use some new piece of technology that will look silly and dated in five years.
Finally, be sure to soften the mom's skin in Photoshop. Duplicate the layer and then go in with the healing brush and basically remove every wrinkle and line from her face. Get in close and really get rid of those lines and blemishes. It should look unnaturally smooth. Then gradually adjust the opacity of that layer so that the original layer starts to show through and you get a nice, softening of her skin that is natural but flattering. You want her to look the way she looks in her husband's mind. And, remember, still photos emphasize flaws because people are frozen and we have all the time in the world to look at their skin. In real life, people move and we see them as living beings, not as rigid statues.
Bottom line, there is nothing wrong with these shots, but you can learn from them for the future.