We only did a day trip to Yala but would have liked to visit more - but there is only so much you can do in 2 weeks.
We made it to Kandy a couple of days after arriving in Colombo, and from there hired a driver for 7 days - with no fixed itinerary he took us wherever we wanted to go and the price was very reasonable. You will often find yourself staying at places recommended by the driver where the owners are either friends/family or he gets free accommodation or free fuel as his cut (I have no problem with that). Every place we stayed as reasonable.
I believe the company we used was Blue Haven.
There is heaps of culture everywhere and I would recommend the usual Dambulla, Polonuwarra, and Buduruwagala (this last one is a bit out of the way and we had to convince the driver to go, probably because of the condition of the road!).