I'm no pro photographer. Instead, I own an accounting firm. However, I have several clients who are photographers and this gives me a unique insight into how they perform financially. As you'd expect, it is a mixed bag. Some do well financially and most enjoy an at least an average income. Others wish they could be doing better. But generally, established photographers tend to have fairly consistent businesses. But it takes many years to get to reach that stage.
The interesting thing is that it is really hard to pick who will be successful financially. Sometimes it is simple things that do well. I've got one client who travels throughout rural Australia taking photos and portraits of people and then blockmounting them. There's nothing really "special" about his photos as they are almost all the same (that being said, they are good). But he's out there every day knocking on doors to get work. But I've seen other people who are exceptional photographers fail commercially. Anyway, the point I'm trying to get across is that you don't know if it is the right time until you give it a try.
If you have a good idea about how your business will run, the type of work you want to target, pricing structure, how you will gain new clients and how you will promote yourself, then you are halfway there. If you have sufficent financial resources to go a couple of lean years (and aren't afraid to risk it) then your chances of being successful increase dramatically. And while the smart thing would be to stay in your current job, realistically, at some point you will need to commit fully to the new business. If you've got everything in place now, why not make the jump?
However, be realistic. Set some objectives based on the income you need to earn. If you are nowhere near where you need to be, reassess the situation. Sometimes, no matter what you do and how hard you try, things just don't work out. Don't be too proud to give up temporarily and try again when you have some different ideas, more capital or the economy is performing better. OK, you might lose some money, but that's probably the worst that can happen. You will gain some business experience. You will gain some new ideas about photography. You will learn what its like to do it professionally. You'll know whether it is or isn't for you. But most of all, you won't be sitting around in twenty years time living in regret wishing you'd have had a go and thinking you're too old, or have too many financial commitments to take the risk.